Competition of pattern forming instabilities due to phase front curvature in an optical system