Ice sheets appeared in the northern hemisphere around 3 Ma (million years) ago and glacial-interglacial cycles have paced Earth's climate since then. Superimposed on these long glacial cycles comes an intricate pattern of millennial and sub-millennial variability, including Dansgaard-Oeschger and Heinrich events. There are numerous theories about these oscillations. Here, we review a number of them in order to draw a parallel between climatic concepts and dynamical system concepts, including, in particular, the relaxation oscillator, excitability, slow-fast dynamics and homoclinic orbits. Namely, almost all theories of ice ages reviewed here feature a phenomenon of synchronization between internal climate dynamics and astronomical forcing. However, these theories differ in their bifurcation structure and this has an effect on the way the ice age phenomenon could grow 3 Ma ago. All theories on rapid events reviewed here rely on the concept of a limit cycle excited by changes in the surface freshwater balance of the ocean. The article also reviews basic effects of stochastic fluctuations on these models, including the phenomenon of phase dispersion, shortening of the limit cycle and stochastic resonance. It concludes with a more personal statement about the potential for inference with simple stochastic dynamical systems in palaeoclimate science.