A broadband, polarization-independent phase modulation spanning the visible range is demonstrated using a polymer/cholesteric liquid crystal composite with optical pitch in the ultraviolet. Polarization insensitivity is achieved as a result of two effects: (1) optical anisotropy of the rod-like molecules is canceled out by the short helical pitch, and (2) stabilization of the Grandjean texture by the polymer network suppresses depolarization. Polarization-independent modulation of the refractive index by approximately 0.045, corresponding to a phase modulation of π at 500 nm, is achieved with submillisecond response times. Our material system opens new avenues for polarization-independent, tunable optical devices, such as narrow bandpass filters, gratings, and adaptive lenses.