We investigate the phase diagram of TmB4, an Ising magnet on a frustrated Shastry-Sutherland lattice, by neutron diffraction and magnetization experiments. At low temperature we find Néel order at low field, ferrimagnetic order at high field, and an intermediate phase with magnetization plateaus at fractional values M/M_(sat)=1/7,1/8,1/9,... and spatial stripe structures. Using an effective S=1/2 model and its equivalent two-dimensional fermion gas we suggest that the magnetic properties of TmB4 are related to the fractional quantum Hall effect of a 2D electron gas.