We show that the inability of a quantum Otto cycle to reach a limit cycle is connected with the propagator of the cycle being noncompact. For a working fluid consisting of quantum harmonic oscillators, the transition point in parameter space where this instability occurs is associated with a non-Hermitian degeneracy (exceptional point) of the eigenvalues of the propagator. In particular, a third-order exceptional point is observed at the transition from the region where the eigenvalues are complex numbers to the region where all the eigenvalues are real. Within this region we find another exceptional point, this time of second order, at which the trajectory becomes divergent. The onset of the divergent behavior corresponds to the modulus of one of the eigenvalues becoming larger than one. The physical origin of this phenomenon is that the hot and cold heat baths are unable to dissipate the frictional internal heat generated in the adiabatic strokes of the cycle. This behavior is contrasted with that of quantum spins as working fluid which have a compact Hamiltonian and thus no exceptional points. All arguments are rigorously proved in terms of the systems' associated Lie algebras.