How cold can I go?

Hi all, for starters I do my surfing in NJ. Being a broke grad student I opted for a 4/3 oneill hyperfreak for my one and only wet suit. I knew it wasn't going to be perfect for every situation and I would possibly have to sit out the coldest days but from what I read a 4/3 would be the most flexible. So far it has served me very well. I put it on the first time when the water dipped below 70ish. Probably would have been okay that day in just board shorts but the suit allowed me to stay in the water for 6 hours straight. As it started to get colder I added 5mm boots and gloves as well as a 3mm vest with a 5mm attached hood. Most recently I was out in 42-45f water (according to surfline) and the air temp was about the same. That day I was out for about 3 hours and got out as warm and comfortable as I went in. So my question is just how much longer will I be able to keep surfing?
According to their website I am already below what they recommend for a 4/3. But I've always been decently tolerant of the cold and I think the vest also helps a good deal. I have ton a ton of searching on the internet and I know this is largely a individual thing but does anyone have else use a vest under their wetsuit? I am just not sure how much warmth it adds. Obviously I can just go try it and see but its about a 3 hour round trip to do that so any insight you could offer would be great.
According to their website I am already below what they recommend for a 4/3. But I've always been decently tolerant of the cold and I think the vest also helps a good deal. I have ton a ton of searching on the internet and I know this is largely a individual thing but does anyone have else use a vest under their wetsuit? I am just not sure how much warmth it adds. Obviously I can just go try it and see but its about a 3 hour round trip to do that so any insight you could offer would be great.