The severe band-limited effect resulted from the low-cost optical transceiver increases the channel memory length and the number of taps of the equalizers. Besides, the interaction of fiber dispersion and square-law detection introduce nonlinear distortions in intensity modulation and direct-detection (IM/DD) transmission systems. The serious band-limited effect and nonlinear distortions degrade the transmission performance and bring challenges to current equalizers for low-complexity implementation. In this paper, we propose a trellis-compression nonlinear maximum likelihood sequence estimation (TC-NL-MLSE) algorithm to compensate the linear and nonlinear distortions with lower complexity. In the TC-NL-MLSE, we introduce a polynomial nonlinear filter (PNLF) to partly compensate both the linear distortions and nonlinear distortions. Then, we establish a look-up-table (LUT) to calculate the nonlinear branch metric (BM). To simplify the calculation, two or three levels with the highest probabilities are selected according to decision thresholds for each symbol to compress the state-trellis graph (STG). This significantly reduces computational complexity on BM calculations especially for high-order modulations. We conduct experiments to transmit beyond the 200-Gb/s PAM-8 signal over 2-km standard single mode fiber (SSMF) at C-band. The TC-NL-MLSE outperforms the reduced-state MLSE with PNLF, and can reach the 7%-overhead hard-decision forward error correction threshold. Moreover, the TC-NL-MLSE reduces the complexity by 97% compared with standard LUT-MLSE, limiting the multipliers around 100 at the expense of only 0.2-dB receiver sensitivity penalty.