The problem is the 3DS market is nowhere near as lucrative as is insisted by many here, or as would be expected by its install base. There is a top-tier of games(YK, Pokemon,MH, core FP titles) that sell really well, a tier below that of major 3rd party offerings that sell around 1 million, and then 85% of its software sells in range that might as well be on Vita. If I were attempting to make a family/kid friendly game I would put it on the 3DS, but it is far from clear that B-tier visual novel or JRPG sells any better there than on the Vita. Look how many 3DS games end in sub-100k, sub-50K, and sub 20K range. Would those clearly have done any worse on Vita. In fact, several of the multiplatform ones do better.
This is factually incorrect.
There are around 50 games between 200k and 1m on 3DS; there are only 10 games above 200k on PSV. There are around 40 games between 100k and 200k on 3DS; and around 20 in the same range on PSV.
PSV showed how it is basically impossible to cross the 300k mark unless you're a fairly big name (God Eater, Persona, Minecraft). Therefore, it is not fair saying that below 1m units, 3DS is basically comparable with PSV. Niche games would sell about the same on both platforms, but this is always true; if a software house would like to reach a sizeable market, PSV is really constrained by its platform and audience (while 3DS userbase is quite heterogeneous, PSV ones is more targeted).
It is also not true that those sub-100k 3DS games would have done better, or the same, on PSV; you also do not count development costs and the fact that 3DS has a better market overseas.
Even a game like Project Mirai was able to outsell a mainline PSV entry, which clearly shows how 3DS is actually a good ecosystem for third parties, while PSV is ok mainly for niche franchises.
For instance, I have trouble seeing DQXI manage to exceed YK/Pokemon at around 2.5 million which would put it substantially below the historical performence of the franchise.
This is simply wishful thinking. Dragon Quest VII sold 1.3m, basically on par with DS remakes, and Dragon Quest Monsters also sold quite well. A traditional mainline 3DS Dragon Quest game would be able to reach at least the same sales Dragon Quest VIII had, thanks to the huge installed base. 3DS showed how it can sustain multi-million sellers; in fact, last year, 5 multi-million sellers games were released in about 6 months. No other platform ever reached that.
DQ is also an exception in that you are looking at the overwhelming proportion of its sales being in Japan. Something like FF or KH which relies on spectacle and has a large Western base pretty much has to be on a console, and none of the handheld spinoffs have done well enough abroad to be able to seriously contemplate a switch.
There is a time issue as well. How long with the 3DS remain viable. Would a major title be viable in late 2016? In Japan perhaps, but abroad?
If I were Square I probably would be putting more support into the 3DS than I am doing(though the lack of Unity makes it troublesome for Mobile ports), and I probably would also have tried to port my Android catalog to Vita. But making 3DS my main platform? God know. It is a one and a half trick pony.
Regarding DQXI, Square has no good options because almost any choice would result in a substantial decline. PS4 maintains the option of Vita/NX. 3DS is basically 3DS, and for a 2017 title that is a risky commitment.
2016 is shaping up nicely for 3DS, both in Japan and abroad. Japan will have Monster Hunter X, The Snack World and Nintendo first party offers such as Mario & Luigi; in the West, Fire Emblem Fates, Yo-kai Watch, and so on.