Manipulation of magnetisation with ultrashort laser pulses is promising for information storage device applications. The dynamics of the magnetisation response depends on the energy transfer from the photons to the spins during the initial laser excitation. A material of special interest for magnetic storage are FePt nanoparticles, for which switching of the magnetisation with optical angular momentum was demonstrated recently. The mechanism remained unclear. Here we investigate experimentally and theoretically the all-optical switching of FePt nanoparticles. We show that the magnetisation switching is a stochastic process. We develop a complete multiscale model which allows us to optimize the number of laser shots needed to switch the magnetisation of high anisotropy FePt nanoparticles in our experiments. We conclude that only angular momentum induced optically by the inverse Faraday effect will provide switching with one single femtosecond laser pulse.