We describe the process of parametric amplification in a directional coupler of quadratically nonlinear and lossy waveguides, which belongs to a class of optical systems with spatial parity-time (PT) symmetry in the linear regime. We identify a distinct spectral PT anti-symmetry associated with optical parametric interaction, and show that pump-controlled symmetry breaking can facilitate spectrally selective mode amplification in analogy with PT lasers. We also establish a connection between the breaking of spectral and spatial mode symmetries, revealing the potential to implement unconventional regimes of spatial light switching through ultrafast control of PT breaking by pump pulses.