Well, if you don't want to sit down and take it, silence is not an option. In the long term, anyway.
Thing is, though, you don't need to speak to be heard. I personally, would completely ignore him. Not just what he says, but his existence completely. If he tried to talk to me, even if he was not trying to be edgy, I would pretend I never heard him. I would not even look into his eyes. That would quickly set the idea that I am, in fact, not cool with him at all.
If things carry on, I'd probably advance to dirty looks. Those sorts of looks you give when you're looking down on people. Like, they're not even worth the shit you stepped in in the morning.
If he's really that persistent, then go on to belittling him. Like, if he asks if you're good at maths, say something like "Yeah, it's useful. Look into it, it's pretty essential in minimum wage jobs" or something like "Why, do you need help?"
I would generally try to piss him off subtly. Though, I am a rather passive person (But I am capable of fighting). I've done it before to people, though, not on the account of racism. Just to people who have given me shit. I am a very petty person, and if it gets to the point where somebody is trying to bully me, I will try to turn it around and bully them. It's childish, I know, but I don't care.
Or if it's just one instance, like if he tries to hug you in a bid to look cool, I'd personally say something like "Wow, you're a cunt."
Succinct and gets the message across. Though, it can get people angry enough to attack you, but oh well, that's just an excuse to punch him.