An analytical model considering modulation-dependent nonlinear effects and second-order interactions between signal and optical amplifier noise is presented for Nyquist-spaced wavelength-division-multiplexing optical communication systems. System performance of dual-polarization modulation formats, such as DP-QPSK, DP-16QAM, and DP-64QAM, is investigated using both the analytical model and numerical simulations. A good agreement between analytical and numerical results shows that, in the case of full-field nonlinearity compensation, accounting for second-order interactions becomes essential to predict system performance of both single- and multi-channel systems at optimum launched powers and beyond. This effect is validated via numerical simulations for signal bandwidths up to ∼1 THz.