Surfing Selfish

I just wanted to get some input on atiitudes in the lineup.
I've been surfing for about 3 years now and started late (40yrs old) so I'm not the best but I do OK... Anyway yesterday the swell was kicking up pretty good but you really had to be in the right place at the rights time. As it goes sometimes, I was not in the right place at the right time most of the day and being as the waves were pumping pretty good it was crowded.
Here's my dilema, there was about 6 waves that I was in perfect position for where I was sitting. As soon as the peak started heading my way, every head in the water paddled over and most of the time, they ended up being in the better postion for the wave. Now, if I go ahead and drop in, I'm the asshole who dropped in when someone else had the inside. Like I said, I'm not the best and most times the shredders who have been surfing since they were 3 pop-up quicker than me and are heading down the line before I'm to my feet.
I had a buddy who said something to this guy that kept paddling over and snaking the prime waves. The guy(and everyone else) seemed to think that there is no problem with it. The attitude seems to be that if you were a better surfer, you'd have the peak and be dropping in first.
So is the trick just to be that selfish surfer that paddles over to the peak no matter who's there and to try for every wave? Do you just go ahead and drop in on the guy who paddled over right next to you from 50 yards down when the peak is coming your way? I've heard it said that surfing is a selfish sport but where do you draw the line? I don't have that super agressive attitude so most times I back out and my wave count suffers for it.
I've been surfing for about 3 years now and started late (40yrs old) so I'm not the best but I do OK... Anyway yesterday the swell was kicking up pretty good but you really had to be in the right place at the rights time. As it goes sometimes, I was not in the right place at the right time most of the day and being as the waves were pumping pretty good it was crowded.
Here's my dilema, there was about 6 waves that I was in perfect position for where I was sitting. As soon as the peak started heading my way, every head in the water paddled over and most of the time, they ended up being in the better postion for the wave. Now, if I go ahead and drop in, I'm the asshole who dropped in when someone else had the inside. Like I said, I'm not the best and most times the shredders who have been surfing since they were 3 pop-up quicker than me and are heading down the line before I'm to my feet.
I had a buddy who said something to this guy that kept paddling over and snaking the prime waves. The guy(and everyone else) seemed to think that there is no problem with it. The attitude seems to be that if you were a better surfer, you'd have the peak and be dropping in first.
So is the trick just to be that selfish surfer that paddles over to the peak no matter who's there and to try for every wave? Do you just go ahead and drop in on the guy who paddled over right next to you from 50 yards down when the peak is coming your way? I've heard it said that surfing is a selfish sport but where do you draw the line? I don't have that super agressive attitude so most times I back out and my wave count suffers for it.