Coherent optical communication systems are still too expensive for metro and short reach networks in the near future. Coherent detection with an intensity modulator replacing a costly dual-polarization in-phase and quadrature modulator could offer a good trade-off between cost and performance for metro networks. The major performance limit of such systems is the nonlinearity. To the best of our knowledge, this Letter offers the first known investigation on nonlinearity of a high speed intensity modulation and coherent detection system using an integrated laser and electro-absorption modulator. Advanced nonlinearity mitigation digital signal processing (DSP) are proposed including a 2×1 multiple-input single-output complex transversal Volterra feed-forward equalizer (FFE) filter, a first proposed post-FFE noise cancellation filter taking into account both pre- and post-cursor noise correlations based on a simple autocorrelation coefficient calculation, and a maximum likelihood sequence estimator. The receiver DSP needs no carrier and phase recovery that is required by a conventional coherent receiver DSP. Results show that, compared to a simple linear FFE equalization, the proposed nonlinear DSP enables over 8 and 4 dB improvement in optical signal to noise ratio (OSNR) sensitivity at 7% overhead hard-decision feed-forward error correction (FEC) and 20% overhead soft-decision FEC threshold bit error rates (BERs), respectively, for a 56 Gb/s intensity modulation and coherent detection system. To the best of our knowledge, a record 18 dB OSNR sensitivity at a hard-decision FEC threshold BER of 3.8×10-3 is also achieved.