Interactions between atoms and light in optical cavities provide a means of investigating collective (many-body) quantum physics in controlled environments. Such ensembles of atoms in cavities have been proposed for studying collective quantum spin models, where the atomic internal levels mimic a spin degree of freedom and interact through long-range interactions tunable by changing the cavity parameters1-4. Non-classical steady-state phases arising from the interplay between atom-light interactions and dissipation of light from the cavity have previously been investigated5-11. These systems also offer the opportunity to study dynamical phases of matter that are precluded from existence at equilibrium but can be stabilized by driving a system out of equilibrium12-16, as demonstrated by recent experiments17-22. These phases can also display universal behaviours akin to standard equilibrium phase transitions8,23,24. Here, we use an ensemble of about a million strontium-88 atoms in an optical cavity to simulate a collective Lipkin-Meshkov-Glick model25,26, an iconic model in quantum magnetism, and report the observation of distinct dynamical phases of matter in this system. Our system allows us to probe the dependence of dynamical phase transitions on system size, initial state and other parameters. These observations can be linked to similar dynamical phases in related systems, including the Josephson effect in superfluid helium27, or coupled atomic28 and solid-state polariton29 condensates. The system itself offers potential for generation of metrologically useful entangled states in optical transitions, which could permit quantum enhancement in state-of-the-art atomic clocks30,31.