Planet formation is generally described in terms of a system containing the host star and a protoplanetary disk1-3, of which the internal properties (for example, mass and metallicity) determine the properties of the resulting planetary system4. However, (proto)planetary systems are predicted5,6 and observed7,8 to be affected by the spatially clustered stellar formation environment, through either dynamical star-star interactions or external photoevaporation by nearby massive stars9. It is challenging to quantify how the architecture of planetary sysems is affected by these environmental processes, because stellar groups spatially disperse within less than a billion years10, well below the ages of most known exoplanets. Here we identify old, co-moving stellar groups around exoplanet host stars in the astrometric data from the Gaia satellite11,12 and demonstrate that the architecture of planetary systems exhibits a strong dependence on local stellar clustering in position-velocity phase space. After controlling for host stellar age, mass, metallicity and distance from the star, we obtain highly significant differences (with p values of 10-5 to 10-2) in planetary system properties between phase space overdensities (composed of a greater number of co-moving stars than unstructured space) and the field. The median semi-major axis and orbital period of planets in phase space overdensities are 0.087 astronomical units and 9.6 days, respectively, compared to 0.81 astronomical units and 154 days, respectively, for planets around field stars. 'Hot Jupiters' (massive, short-period exoplanets) predominantly exist in stellar phase space overdensities, strongly suggesting that their extreme orbits originate from environmental perturbations rather than internal migration13,14 or planet-planet scattering15,16. Our findings reveal that stellar clustering is a key factor setting the architectures of planetary systems.