A dilatometer cell that can detect sub angstrom changes in the length of solid specimens in the temperature range 5 K<T<350 K is described. It is constructed entirely from fused quartz (silica). This is a capacitive-based technique, where the spacing between two metallized plates allows detection of changes in length. It is also a relative measurement in the sense that the capacitor plate spacing is determined by the relative thermal expansions of the cell and sample. The advantage of using fused quartz is that it has among the smallest known thermal expansions. As a result, for most measurements, the sample's thermal expansion dominates the capacitance change. The construction and performance are described. It is directly compared to a dilatometer cell constructed of copper. Some thermometry and temperature control issues specific to dilatometry measurements are discussed.