I've read alot about this question and the answer is generally not about the boat style or size but about decision making of the captain. ie picking the right weather to go out in and knowing when to come home - that makes sense.
I've been "offshore", for me up to 20nm, in 15-17 footers (plastic and aluminium) in 10-20 knots winds swells to 1-1.5m and never really felt unsafe. Yep just slow going.
The reason for the question is that price and trailer space are limiting my selection of boat and am looking at tiller steer 4 - 4.5m plastic, inflatable rib (very buoyant, stable, flat self draining decks) or less keen on a tinnie (less stable, not as family friendly when insheltered waters, expensive and heavy for a sealed floor etc but open to opinions) for 10-15K new but happy to look at second hand (which is tough these days).
Will be used mostly in Moreton bay and near reefs around Sunshine and Gold coast etc. Ideally out to the close islands camping further north (ie Keppels and up to Whitsundays). I've fished from inflatables before. Not as ideal but workable.
Its easy to say I'll only go out in good weather but I'm trying to be ready for time when the weather unexpectedly turns nasty or overnight goes bad if I'm up north.
Maybe a better question would be..... what is the minimum size boat that you would not be pissed off saving if the weather goes bad ie you were an idiot going out in that boat or that its just bad luck mate.
Pete.