We report the direct integration and efficient coupling of nitrogen vacancy (NV) color centers in diamond nanophotonic structures into a fiber-based photonic architecture at cryogenic temperatures. NV centers are embedded in diamond micro-waveguides (μWGs), which are coupled to fiber tapers. Fiber tapers have low-loss connection to single-mode optical fibers and hence enable efficient integration of NV centers into optical fiber networks. We numerically optimize the parameters of the μWG-fiber-taper devices designed particularly for use in cryogenic experiments, resulting in 35.6% coupling efficiency, and experimentally demonstrate cooling of these devices to the liquid helium temperature of 4.2 K without loss of the fiber transmission. We observe sharp zero-phonon lines in the fluorescence of NV centers through the pigtailed fibers at 100 K. The optimized devices with high photon coupling efficiency and the demonstration of cooling to cryogenic temperatures are an important step to realize fiber-based quantum nanophotonic interfaces using diamond spin defect centers.