We use molecular dynamics simulations to compute the spatially resolved static dielectric constant of water in cylindrical and spherical nanopores as occurring, e.g., in protein water pockets or carbon nanotubes. For this, we derive a linear-response formalism which correctly takes into account the dielectric boundary conditions in the considered geometries. We find that in cylindrical confinement, the axial component behaves similar as the local density akin to what is known near planar interfaces. The radial dielectric constant shows some oscillatory features when approaching the surface if their radius is larger than about 2 nm. Most importantly, however, the radial component exhibits pronounced oscillations at the center of the cavity. These surprising features are traced back quantitatively to the non-local dielectric nature of bulk water.