rank is used for defining rank constraints.
r = rank(x)
The nonlinear operator rank is used to add rank constraints in SDP problems.
F = [F, rank(X)<=1];
The variable returned from the rank operator can currently only be used for bounding ranks from above on symmetric variables, and is only supported by the solver LMIRANK. See the tutorial on rank constrained SDPs