We apply intense terahertz (THz) electromagnetic pulses with field strengths exceeding 2 MV cm(-1) at ~1 THz to window and substrate materials commonly used in THz spectroscopy and determine the induced optical birefringence. Materials studied are diamond, sapphire, magnesium oxide (MgO), polymethylpentene (TPX), low-density polyethylene (LDPE), silicon nitride membrane (SiN) and crystalline quartz. We observe a Kerr-effect-type transient birefringence in all samples, except in quartz and Si, where, respectively, a linear electrooptic signal and a response beyond the perturbative regime are found. We extract the nonlinear refractive indices and the electrooptic coefficient (in the case of quartz) of these materials and discuss implications for their use as windows or substrates in THz pump-optical probe spectroscopy.