Contemporary gravitational-wave detectors are fundamentally limited by thermal noise-due to dissipation in the mechanical elements of the test mass-and quantum noise-from the vacuum fluctuations of the optical field used to probe the test-mass position. Two other fundamental noises can in principle also limit sensitivity: test-mass quantization noise due to the zero-point fluctuation of its mechanical modes and thermal excitation of the optical field. We use the quantum fluctuation-dissipation theorem to unify all four noises. This unified picture shows precisely when test-mass quantization noise and optical thermal noise can be ignored.