Data from the Gravity Recovery and Interior Laboratory (GRAIL) mission have revealed that ~98% of the power of the gravity signal of the Moon at high spherical harmonic degrees correlates with the topography. The remaining 2% of the signal, which cannot be explained by topography, contains information about density variations within the crust. These high-degree Bouguer gravity anomalies are likely caused by small-scale (10's of km) shallow density variations. Here we use gravity inversions to model the small-scale three-dimensional variations in the density of the lunar crust. Inversion results from three non-descript areas yield shallow density variations in the range of 100-200 kg/m3. Three end-member scenarios of variations in porosity, intrusions into the crust, and variations in bulk crustal composition were tested as possible sources of the density variations. We find that the density anomalies can be caused entirely by changes in porosity. Characteristics of density anomalies in the South Pole-Aitken basin also support porosity as a primary source of these variations. Mafic intrusions into the crust could explain many, but not all of the anomalies. Additionally, variations in crustal composition revealed by spectral data could only explain a small fraction of the density anomalies. Nevertheless, all three sources of density variations likely contribute. Collectively, results from this study of GRAIL gravity data, combined with other studies of remote sensing data and lunar samples, show that the lunar crust exhibits variations in density by ±10% over scales ranging from centimeters to 100's of kilometers.