Just to be clear on where I'm coming from, having downtime in an RPG is absolutely necessary. You can't have nonstop excitement and action because it becomes tiresome and you grow numb to it. Having slower moments where you can take a breather, take in the story, setting and characters, is an important part to any good narrative. The issue comes from how FFXVI handles the downtime.
You spend much of your downtime in FFXVI going back to the Hideaway, talking to NPC after NPC after NPC, going to a location on the map, talking to NPC after NPC after NPC, go kill something, return to Hideaway for more talking, so on so forth. Most of these cutscenes during this time are handled the same way. Clive and the NPC stand statically in front of each other, and exchange long winded dialogue. You do this for about 3-5 hours depending, before you get to the next heavy story section of the game. This is where higher quality cutscenes kick in with good camerawork, directing, facial and body animations. A dungeon is involved, and it all leads to a few boss fights including a big set piece fight. After this, you repeat the same exact cycle of downtime, with the same approach to how it's executed. The high octane action and set pieces of FFXVI are on the complete opposite end of the spectrum of the very slow and heavy dialogue sections of the Main Story. To add, let's say you need a break from all the talking and just want to explore. Exploration is very limited, there's not much to do outside of Sidequests which take the same approach to the Main Story downtime, and you can only unlock 2-3 Hunts at a time by progressing the story. So then you very quickly circle back to continuing with the Main Story once you realize how barebones the side content is.
All of the above is what shines a light on the pacing in FFXVI for many folk. Look at the Prologue of FFXVI. There's way more cutscenes and talking than there is gameplay. But look at the way the dialogue and cutscenes are handled during those first two hours. It looks like a movie. Scenes are shot in ways that are interesting, there's a lot happening on screen, and you're moving from one location to another. Now juxtapose that with the rest of the game where so much of your time is spent in the Hideaway with Clive and an NPC standing in front of each other, always using the same camera angles, and dialogue that goes on longer than it should.
I'll close out with this. Look at Final Fantasy X, another entry in series that is linear. You have lots of quiet downtime moments in that game, but observe the way those moments are handled. You're always in a new location than you were before, you're always learning more about the world of Spira and the characters you travel with, and there's always a point to them. Those moments say far more, using less time. And if you like the way FFXVI handles the downtime then god bless. I don't want to rain on your parade if you enjoyed everything about the story. As someone who likes FFXVI but has issues with it, I'm just trying to elaborate why the pacing of the game is a big issue for people.