The problem isn't "big numbers", it's mindshare and market share erosion.
Stagnant platforms tend to lose support long term from both consumers and developers. If you need evidence of this, look at what's happening to the Vita and Wii U, and how the PS4 is now joining them.
Will the 4DS lose as much support? I would say no as it's the only platform left, but each drop continues to be painful.
If we compare the breadth of large software the system had announced in its first year to now, and then to the DS and PSP as well, we see quite a few major publishers condensing down to a much smaller, safer lineup as they use their other resources to chase the booming mobile industry (Square, Capcom, and Konami to just name three). This isn't good for a platform's long term health. That was true for the PS3 as it lost a variety of mid tier games, true for the PSP as it collapsed in the West, true for the Wii as third parties abandoned it worldwide, and will be true for the 3DS, which we can see by just looking at how it's selling right now.
The idea of an early 4DS lies upon the assumption that Nintendo can create something more appealing to both consumers and developers than the 3DS. If that's not true, then you're right, they're largely intractably fucked and it doesn't really matter when they launch so they might as well quadruple down on QoL and hope it gives them something resembling a strong revenue stream, though that feels like quite the longshot itself.