To paddle or to bail

I already know what most people are going to say to this question, that it is just experience but I'll ask anyway.
When the waves get a bit bigger, bigger for me being 5-6ft + I always seem to hesitate when looking down at the big drop in front of me, normally i paddle as hard as i can but I still feel I could do with more speed, (believe me I have watched every single video ever made for paddling tips) but as the wave comes under me and you start to point downwards something in my brain says, 'what, you want to paddle downhill, are you crazy' then I either try pop up too soon or the waves goes right by.
Last night i committed myself more and took a few more paddles facing downhill and making sure the wave has caught the board before popping up and caught me a couple of awesome waves.
But my question is, how do you know when to keep paddling down the face of the wave or abandon as I just think I am going to get caught by the lip and thrown towards the beach.
I Imagine it depends on the kind of waves, ect but is the best thing just to go for it anyway and you'll learn eventually or is there any tips?
I ride a 6.0 37lt swallow tail or a 6.1 31lt squashed tail.
When the waves get a bit bigger, bigger for me being 5-6ft + I always seem to hesitate when looking down at the big drop in front of me, normally i paddle as hard as i can but I still feel I could do with more speed, (believe me I have watched every single video ever made for paddling tips) but as the wave comes under me and you start to point downwards something in my brain says, 'what, you want to paddle downhill, are you crazy' then I either try pop up too soon or the waves goes right by.
Last night i committed myself more and took a few more paddles facing downhill and making sure the wave has caught the board before popping up and caught me a couple of awesome waves.
But my question is, how do you know when to keep paddling down the face of the wave or abandon as I just think I am going to get caught by the lip and thrown towards the beach.
I Imagine it depends on the kind of waves, ect but is the best thing just to go for it anyway and you'll learn eventually or is there any tips?
I ride a 6.0 37lt swallow tail or a 6.1 31lt squashed tail.