Is the game really that bad? I've heard people criticize it over the years often but it still seems like a good game. How do you think it compares to The Legend of Zelda: Four Swords Adventure, if you've played?
Yes, it is that bad. It is incomparable to Four Swords Adventures, because that game uses the basic 2d Zelda template gameplay, which is in and of itself enjoyable, and it also had quite charming 2d art.
On the other hand, Phantom Hourglass was made in a specific time for Nintendo, one where the success of the DS meant that they were forcing touch controls on every game series to justify that method of input (we're pre-iPhone here), and also trying to "blue ocean" every game series so that their expanded market of moms and seniors could enjoy it too. They also, for certain titles, wanted to prove that the DS could do 3d graphics competently, despite the hardware really not being strong enough.
The result of all of these design influences in Zelda: PH was a really mediocre game, which by Zelda standards was a disastrous game.
The touch controls, which were praised at the time for their noveltly, are not good. The best way you can tell this is by the actions of enemies. Even a basic chu-chu will wobble around you harmlessly - not attacking because the designers have no confidence that you will successfully be able to swing your sword when you need to. This problem recurs throughout the game. Areas where the touch screen works better such as drawing a boomerang path or firing a bow are still simplistic, and don't really provide more fun than the immediate reaction of a button press. You constantly feel like you'd be better off using buttons and just using the touchpad from time to time.
The visuals are horrid. Everything is sub-N64 in both polygons and textures, and the cel-shading can't fully make up for that. Worse, the level design is restricted to almost entirely straight lines and large areas, and never feels organic like the Wind Waker did.
The story is just about as bad a continuation from Wind Waker as there could have possibly been - hey, let's take the most charismatic character and turn her into a statute 30 seconds in. Awful. Linebeck is good, but he can't carry the game.
And worst of all, the game requires you to return to the same tower 4 or 5 times during the game. It's bland and boring throughout (like all of the dungeons).
It's a terrible game. The true miracle of Zelda on the DS is that Spirit Tracks took this ruinous foundation and actually made something reasonably fun out of it.
But I thought at the time and still do now that if they had just stuck to sprite designs and normal controls for Zelda they could have made classics on the DS - in the same vein that Castlevania did. But Nintendo's philosophy towards that machine is why it's remembered as a 3rd party powerhouse but a 1st party wasteland.