Lack of compelling software coupled with large existing back catalogs.
I think this can be summarily described as generational fatigue.
If we consider the Wii a "fad" because of its quirky controller -- which people tried, played with, had fun with, then discarded once they no longer found the newness compelling -- then the 360 and PS3 are similarly faddish, just in a way we accept as normal because we are accustomed to it. People try new systems because of their awesome new graphical and technical capabilities, they play the new games with the best graphics and largest worlds, and then gradually grow accustomed to these graphics. The newness of the graphics and worlds, like the newness of the controller, ceases to be inherently exciting. Customers become bored and move on in exactly the same way some Wii gamers do, it's just that the 360/PS3 had huge third party support to sustain this "fad" for a longer period of time.
In other words, the active install base is decreasing as new users aren't added fast enough to replace those that are buying far less games or who have abandoned their consoles entirely because they bought new shiny things or decided to exit the market entirely for now.