We introduce an all-glass, vacuum tight, fiber-integrated and alkali compatible spectroscopy device consisting of two conventional optical fibers spliced to each end of a capillary. This is mainly realized through a decentered splicing method allowing refilling of the capillary and controlling the vapor density inside. We analyze the light guidance of the setup through simulations and measurements of the transmission efficiency at different wavelengths and show that filling it with highly reactive alkali metals is possible, and that the vapor density can be controlled reliably.