We have studied the hole charging spectra of self-assembled InAs quantum dots in perpendicular magnetic fields by capacitance-voltage spectroscopy. From the magnetic-field dependence of the individual peaks we conclude that the s-like ground state is completely filled with two holes but that the fourfold degenerate p shell is only half filled with two holes before the filling of the d shell starts. The resulting six-hole ground state is highly polarized. This incomplete shell filling can be explained by the large influence of the Coulomb interaction in this system.