Can I solve BMIs without PENBMI or PENLAB?

Yes (and you will have to as these solvers typically do not work well).

Develop your own heuristic (fix variables, alternating coordinates, trust regions etc). This can easily be done in high-level YALMIP code.

Alternatively, try the global solver BMIBNB which supports nonlinear semidefinite constraints. Most likely it worn’t work as your problem is too large, but it never hurts to try.