Well the first thing, Fast Eddie, is to not be so fast! Ah forgive me , I just had to say that.
Mate, I am not qualified in the slighest to comment on the Raby Bay area as I usually go from Manly out into the open Bay or inside the islands up to Mud Island and for that matter, I am just a five year beginner boatie although I do hit the water 2 - 6 times each week. (14 months long service leave, you see)
However, I usually hestitate for a long while before I will go out in conditions that are forecast to be greater than 15 Knots. None of the forecasting services are reliable but I am finding that Seabreeze.com.au is not bad when the weather is relatively stable over a a few days. When it is not so stable, I forget Seabreeze and take note of the BoM. I reiterate that none of these are totally reliable, as I have been caught out at night on my lonesome in the bay when the conditions changed from 6 knots to 30 knots within a half hour and none of the weather services predicted that change.
Also, in summer, the best boating time is invariably the mornings. So I usually plan my return trip at around 11.15am and then look at the Seabreeze.com.au "Inner Beacon" wind graphs to see if I was right in coming in at that time. I am rarely wrong on that score.
Thirdly, a following sea is going to be a lot easier to come home in than a head on sea that you have to punch into. Thus if you are heading north towards Raby Bay from the southern side of Peel or other points south into the northerlies that we have been getting, you will cop lumpy conditions, especially if you have tide against wind.
Not sure if these comments help - there are a lot more experienced boaties around than me - but these comments might illicit more qualified perspectives.