moving away from the peak

I usually surf beach breaks and breakwater/jetties. I've been working on trying to keep myself close to the peak rather than taking off on the shoulder but I always find myself drifting away from it. Usually before I paddle out I look where the waves are initially breaking then paddle to that location and try to align that area with something on land (sign, tree, etc.) Then I as I'm sitting I see waves peak elsewhere. Basically I never seem to be in the right spot. I'm not sure if it's the current that is moving me away from the peak or just plain bad luck. I notice other surfers just sit and happen to be in the right spot.