This means that you have used so called nonlinear operators to model your problem, and most likely you have defined a problem which cannot be represented using standard convex constraints. If you know that your model is convex, try to model the nonlinear terms by hand to see if you actually are correct (the convexity analysis is conservative, although in most cases failure in the convexity propagation is due to actual nonconvexity.
You May Also Enjoy
Working with polynomials, function values, derivatives, integrals and their properties
Minor fixes and improvements
How do I create a cheap Ferrari?
YALMIP creates the options structure for FMINCON through the call fmincon(‘default’) which creates a different set of options compared to the alternative me...