There are clear public policy reasons to worry about polyamory that simply don't apply to homosexual unions. Legalizing gay marriage probably isn't going to turn everyone gay; it's a lot more plausible that, over time, normalized polyamory will produce lots more polygamy than polyandry to the detriment of sexual equality and with unfortunate side effects similar to what China sees as a result of widespread sex-selective abortion. This isn't even that speculative - polygamy has been widespread before, and societies with lots of polygamy have had lots of problems because of it. Polyamory may not be in-principle problematic, but we have a lot more work to do on sexual equality before we try to normalize it. There is no similar reason to worry about legitimizing homosexual unions; it's ridiculously implausible that some huge percentage of the population will turn gay as homosexuality becomes less stigmatized, and it's likewise implausible that there will be a large and problematic imbalance of openly gay men to openly gay women. It's much easier to make the rational basis argument here.
Also, the polyamorous relationships that would actually seek legal recognition would be the ones that we want to discourage. There are plenty of harmless polyamorous relationships, but the ones that would jump at legal plural marriage are often abusive.
Edit: And the psychological slippery slope argument - the idea that making marriage no longer "special", in a very particular sense, will lead to people becoming more inclined to accept kinds of sexual relationships that we all think are problematic - is silly because it cuts both ways. If your position is that it's really, really important that everyone think that particular sorts of relationships are better than others and that it's really, really important to have a special institution for this, then you should be really, really concerned with making sure that the sorts of relationships that people do think are better than all others are in that institution. The institution of marriage does in fact face a crisis of legitimacy. Something like 70% of young people (the people whose attitudes towards marriage we're most concerned about) think that marriage as it exists in most of the US is a bigoted institution. People with stable, loving relationships who want to be married are told that they can't get married, and are instead forced to model admirable unions that aren't marriages. I'm having a hard time imagining a politically feasible policy which would do a better job of blurring the lines between the relationships we deem marriages and those we don't. Unless gay marriage is problematic in itself - and people making this sort of slippery slope argument are typically making it precisely because they don't want to argue against gay marriage directly - there's every reason to quickly move all of these unproblematic gay relationships into the "marriage" category so that we can all go back to uncontroversially sneering at other sorts of relationships. Don't fight a battle where winning means convincing lots of people that "marriage" just isn't that big of a deal. It is absolutely bizarre that the mainstream conservative argument against gay marriage is that it's perfectly acceptable for two people who love each other and who may even want to raise children together to remain unmarried.