Dude yeah you were trashed and he left you alone with his sister. I'm not saying it was right but it's not something to dish out the pain for. I don't see any reason to have some formal fight or anything, violence is uncivilized, so there is no point in trying to make it civilized by pretending it has rules.
Obviously you don't want to fight, so I say continue with your life and keep your eye out. If he attacks you, first try to escape and maybe get somewhere with eye witnesses. But if you get trapped and have to fight, treat him like any random attacker, do what it takes to defend yourself.
I have no problem with going for eyes, neck, balls, biting, weapons, whatever (Jeet Kun Do has good principles about this). Violence isn't something reasonable people do, if a person is resorting to it with you then you can't be sure how far they'll go, so it's a matter of survival.
Don't go further than you must to stop him, but do what it takes to stop him at least enough to get away. If you can get away with minimal contact altogether then thats even better. You are not indebtted to have a "fair" fight with him, don't let some unwritten rules get you possibly injured in a way you'll have to live with for the rest of your life (it IS very possible in adult fights).