How do you deal with Idiots in the water?

I was out surfing today when this older local guy wanted an argument, accusing me and my friend of not waiting in the line up which was completely rubbish.
I think I am actually one of the best guys out there for giving way to other surfers and keeping out of the way, at my local break no one actually waits in a line up anyway which is annoying, its normally so busy and they paddle around each other like crazy, the best surfers always catch the waves and I catch anything not already taken.
We paddled out and I was in the perfect spot for this wave which was about to break, no one was even close to catching it so i started paddling, when I was about to jump up this guy just screams something out as loud as he could which made me stop as I thought I must have missed someone already riding the wave, this guy was far behind to our right with not even the smallest chance of getting it, he wasn't even paddling for it, we didn't even think he was screaming at us to start with.
About 30 minutes later he then started being very aggressive saying he is a local and we must not know the rules, even though we have surfed this spot for 3 years, I think he was just pissed as he didn't catch any waves, I did start telling him we did nothing wrong but of course this made him more aggressive.
How do you guys deal with situations like this? its so hard just to nod along and ignore idiots like this but I know its the best thing to do.
I think I am actually one of the best guys out there for giving way to other surfers and keeping out of the way, at my local break no one actually waits in a line up anyway which is annoying, its normally so busy and they paddle around each other like crazy, the best surfers always catch the waves and I catch anything not already taken.
We paddled out and I was in the perfect spot for this wave which was about to break, no one was even close to catching it so i started paddling, when I was about to jump up this guy just screams something out as loud as he could which made me stop as I thought I must have missed someone already riding the wave, this guy was far behind to our right with not even the smallest chance of getting it, he wasn't even paddling for it, we didn't even think he was screaming at us to start with.
About 30 minutes later he then started being very aggressive saying he is a local and we must not know the rules, even though we have surfed this spot for 3 years, I think he was just pissed as he didn't catch any waves, I did start telling him we did nothing wrong but of course this made him more aggressive.
How do you guys deal with situations like this? its so hard just to nod along and ignore idiots like this but I know its the best thing to do.