Astro,
Don't give up mate, I was in a similar position 3 years ago and persistance pays off. Two of my off shore PBs have come from plastics, snapper & mahi mahi. I have found SP work better early to mid morning where we normally fish.
I still fish with bait at some stage most trips out, around 30% bait, 70% SP. I gave the PE jigs a go last trip so my bait rod didn't get touched (although when using the PE jigs a bit of bait was put on the hook).
You just have to suck it up and cop a bagging from your mates when the SP don't work...but hang in there and you will be able to give it back when you land that trophy fish!!
Adam