Combining constraints from microlensing and Lyman-α forest, we provide a simple argument to show that large spatial clustering of stellar-mass primordial black holes at the time of formation, such as the one induced by the presence of large non-Gaussianities, is ruled out. Therefore, it is not possible to evade existing constraints preventing stellar-mass primordial black holes from being a dominant constituent of the dark matter by boosting their initial clustering.