While microgels are in general described as soft particles, polystyrene (PS) microgels can be synthesized in a way that cross-link density has only a minor influence on their physical properties. Even though the particles swell in a good solvent, the imparted slight softness still allows a mapping on hard sphere behaviour for a large range of cross-link densities [Schneider et al., Soft Matter, 2017, 13, 445]. Nevertheless, the hard sphere analogy breaks down as soon as polymer chains are added to these systems. Quantitative differences between PS microgels and true hard sphere systems appear and the differences between stronger and weaker cross-linked PS microgels can be observed. To gain deeper insights into the origin of these deviations from true hard sphere behaviour, this work is addressed to a systematic study of the colloid-polymer interactions in PS microgel-polymer mixtures. We investigated the aggregation behaviour (namely aggregation concentration and cluster structure) as a function of colloid size, cross-link density and colloid-polymer size ratio in very dilute colloidal suspensions. Our results show that the interplay of cross-link density and polymer size is a key parameter for the strength of the colloid-polymer interactions. Furthermore, the centre-to-centre distance of the colloidal particles in the formed clusters decreases if the cross-link density is decreased, allowing for a higher packing density. This may also explain the unusually high fluid packing fractions observed in the re-entry region of the phase diagram of PS microgel-free PS polymer mixtures.