Soul Sacrifice for PSVita (My favorite handheld game ever with some of the most fun uses of magic I have seen used in a game. Never was ported from the Vita sadly.
Freedom Fighter- Also a Monster Hunter like game on VIta that was never ported. I really loved the combat and the time counter mechanic. The grappling system was awesome and reminded me of Attack on Titan.
Dreams- One of the most ambitious games I have witnessed. I think its niche nature and being stuck on consoles really held it back from being more popular. It has tons of potential and hope it can be ported to PC one day for people who want finer controls.
The Last Guardian- Went from overhyped, to severely underrated. It had a lot of criticisms of its performance, camera, and controls (Amazing how another recent game had a lot of the same issues and was labeled a masterpiece, but I will keep my salt out of it.) Aside from these issues, this game was a technical marvel. The A.I. of Trico was incredible, and the animations and physics were great. When the boy walked, you can feel the weight with how he moved, fell, and lifted objects. When Trico jumped in the water, the water pushed you around due to the games use of physics.
Knights in the Nightmare- I never finished it, but I really enjoyed the gameplay. What held it off from being popular was it was very niche with a complex combat system that combined bullet hell and turn based strategy. I thought it was a pleasant blend of mechanics after you got the hang of it. I can also see permadeath warding off people as well.
Pokémon Mystery Dungeon Explorer series- Imo, these are way more enjoyable than the main series in almost every way from story, gameplay, depth, and tone. They were rated so low from journalist and never got anywhere near the popularity of the main series. I will also throw in Pokémon Colosseum/XD in there too.
Absolver- The younger, multiplayer version of Sifu (kinda). A martial arts-based game where you could customize your movesets and character in an open environment and team up with other players. It was what I always wanted the fighting game genre to evolve into. Sadly, it didn't gain much traction.
Advance Wars Days of Ruin- The best game in the series, imo. My favorite part was the more serious tone and the promotion system. Sadly, it wasn't as successful and was the last main Advance Wars game since 2008.
Neo: TWEWY- The long-awaited sequel to one of the most beloved handheld games of last decade had some of the most lukewarm attention when it came out. In some ways it was even better than the first game and it is almost like an afterthought. I hear people mention how Square hasn't released a good game in years and want to scream.
Astral Chain- Aside from some of the horrible platforming, it has some really satisfying combat and presentation. Sadly, it wasn't a big success so now we get games like Babylon's Fall from Platinum. I hope Bayonetta 3 hits it out of the park.
Dragon's Crown- An excellent side scrolling beat em up with the gorgeous Vanillaware artstyle they are known for. Aside from the blatant fanservice, Vanillaware games are usually fairly niche.
Fantasy Life- I would kill for a non-mobile sequel. One of the better life sims games I have played.
Dragon Quest Monsters: Joker Series and World of Final Fantasy- Better monster collecting games than Pokémon, imo and are severely overlooked.