Dealing with aggro in the water.

Hi guys, I'm not sure to how many of you this will apply, but the beach I surf out gets extremely crowded, not just with surfers, but swimmers, kayakers, boats, it's a bit of a mind field out there.
There are certain rules in place to keep people safe - ie swimmers stay between the flags, no fibre glass boards at the north end, no surfing soft tops in the flags etc. (See beach flags guide)
Anyway this morning my friend and I were out at the one breaking bank had flags in the middle of it, so we went to the side of it - this is allowed in north bondi, as long as you're not on a fibre glass board. And just to be clear we were not in the flags, nor were we drifting into them. Anyway I'm out back and I hear my friend being abused by this geriatric swimmer - who's calling her a blow in and saying she can't have her board there, and "I've lived here all my life and don't you tell me what the rules are" - she's very placid, had just pointed out she wasn't in the flags and was just sitting there while this guy goes off.
I am not very placid and I hate localism at the best of times, but on my own beach! (I've lived there for ten years, sadly I'm not 923 so yeah, the old geezer did have some years on me) I paddled over to defend my friend and pointed out that we had every right to be there and that having lived there for 10 years I was as much a 'local' as he was.
At this point he told us both to 'fleshin' off back to North America or wherever you came from' - if either of us hadn't been white I would have punched him at that point, but instead I just told him he was about 8000 miles off, slowclapped him and paddle away.
But it rattled us and ruined the start of the session, I surf to get away from aggro bull not to get embroiled in it. It' s not the first time, I've come across these guys as a group - most of the time it's laughable - a group of them hung out the back once, treading water, just so they could heckle surfers! Seriously don't they have anything better to do with their time?
So rant over, how do you guys deal with aggression in the water?
There are certain rules in place to keep people safe - ie swimmers stay between the flags, no fibre glass boards at the north end, no surfing soft tops in the flags etc. (See beach flags guide)
Anyway this morning my friend and I were out at the one breaking bank had flags in the middle of it, so we went to the side of it - this is allowed in north bondi, as long as you're not on a fibre glass board. And just to be clear we were not in the flags, nor were we drifting into them. Anyway I'm out back and I hear my friend being abused by this geriatric swimmer - who's calling her a blow in and saying she can't have her board there, and "I've lived here all my life and don't you tell me what the rules are" - she's very placid, had just pointed out she wasn't in the flags and was just sitting there while this guy goes off.
I am not very placid and I hate localism at the best of times, but on my own beach! (I've lived there for ten years, sadly I'm not 923 so yeah, the old geezer did have some years on me) I paddled over to defend my friend and pointed out that we had every right to be there and that having lived there for 10 years I was as much a 'local' as he was.
At this point he told us both to 'fleshin' off back to North America or wherever you came from' - if either of us hadn't been white I would have punched him at that point, but instead I just told him he was about 8000 miles off, slowclapped him and paddle away.
But it rattled us and ruined the start of the session, I surf to get away from aggro bull not to get embroiled in it. It' s not the first time, I've come across these guys as a group - most of the time it's laughable - a group of them hung out the back once, treading water, just so they could heckle surfers! Seriously don't they have anything better to do with their time?
So rant over, how do you guys deal with aggression in the water?