Mapping the dielectric constant at the nanoscale of samples showing a complex topography, such as non-planar nanocomposite materials or single cells, poses formidable challenges to existing nanoscale dielectric microscopy techniques. Here we overcome these limitations by introducing Scanning Dielectric Force Volume Microscopy. This scanning probe microscopy technique is based on the acquisition of electrostatic force approach curves at every point of a sample and its post-processing and quantification by using a computational model that incorporates the actual measured sample topography. The technique provides quantitative nanoscale images of the local dielectric constant of the sample with unparalleled accuracy, spatial resolution and statistical significance, irrespectively of the complexity of its topography. We illustrate the potential of the technique by presenting a nanoscale dielectric constant map of a single bacterial cell, including its small-scale appendages. The bacterial cell shows three characteristic equivalent dielectric constant values, namely, εr,bac1 = 2.6 ± 0.2, εr,bac2 = 3.6 ± 0.4 and εr,bac3 = 4.9 ± 0.5, which enable identifying different dielectric properties of the cell wall and of the cytoplasmatic region, as well as, the existence of variations in the dielectric constant along the bacterial cell wall itself. Scanning Dielectric Force Volume Microscopy is expected to have an important impact in Materials and Life Sciences where the mapping of the dielectric properties of samples showing complex nanoscale topographies is often needed.