Hole transport experiments were performed on a gated double quantum dot device defined in a p-GaAs/AlGaAs heterostructure with a single hole occupancy in each dot. The charging diagram of the device was mapped out using charge detection confirming that the single hole limit is reached. In that limit, a detailed study of the two-hole spin system was performed using high bias magnetotransport spectroscopy. In contrast to electron systems, the hole spin was found not to be conserved during interdot resonant tunneling. This allows one to fully map out the two-hole energy spectrum as a function of the magnitude and the direction of the external magnetic field. The heavy-hole g factor was extracted and shown to be strongly anisotropic, with a value of 1.45 for a perpendicular field and close to zero for an in-plane field as required for hybridizing schemes between spin and photonic quantum platforms.