Yeah that's what I thought would happen with GW Bush. It lasted only a a midterm and Obama's elections. Even though Bush did way worse things than Obama and handed him the great recession, people forget.
To be fair, that probably had a lot more to do with it being a midterm election.
Midterms are usually bad for the White House incumbent party because the out party's base is hot and bothered and this is the most obvious way to hit back, while much of the incumbent party's marginal base is either complacent, disappointed, or ignorant of what's at stake. Breaking the tie are independents who are often just unhappy with whoever's in charge, and the presidency is the most visible.
If voters betrayed Obama, it wasn't in 2012 - we net gained Senate seats in a very difficult year, Obama won comfortably (NC was the only state contested by either party that he lost), and if not for gerrymandering we probably would have won the House majority too.