det overloads determinant in two different ways
y = det(X) y = det(X,'polynomial')
By default, YALMIP will create a callback operator to model the determinant. This means that function values and derivatives are computed when the solver asks for these, and that you thus have to use a callback based solver (nonlinear solvers such as FMINCON, IPOPT and KNITRO).
Alternatively, you can ask YALMIP to derive the symbolic polynomial expression of the polynomial. This will enable the use of PENLAB and PENBMI. Note that the polynomial representation of the determinant has horrible complexity and only can be used on very small and/or sparse matrices.