I grew up in Victoria and my grandfather (rest his soul) always said -
"When the wind's in the west the fish bite best, when the wind's in the east the fish bite least".
Having said that he and I would fish in all conditions (and I still do) because another of his sayings was -
"Anytime is a good time to go fishing - catching a fish is a bonus!"
Pete