[Mockery]

In all seriousness, you're not a pussy for not being able to fight back against a guy. Just be brutally honest with the guy, not to where you insult him or anything, just say that you're too busy and you're really not interested in getting drunk with him. Tell him to go to a frat party or something, maybe that'll get him out of your hair.

If he still persists and ignores your requests to leave you alone, either report him or start making his life miserable in whatever ways you can come up with until it gets through his thick skull.