Phase transitions in the early Universe can readily create an observable stochastic gravitational wave background. We show that such a background necessarily contains anisotropies analogous to those of the cosmic microwave background (CMB) of photons, and that these too may be within reach of proposed gravitational wave detectors. Correlations within the gravitational wave anisotropies and their cross-correlations with the CMB can provide new insights into the mechanism underlying primordial fluctuations, such as multifield inflation, as well as reveal the existence of nonstandard "hidden sectors" of particle physics in earlier eras.