An optical approach to cancel the radio frequency self-interference for an in-band full-duplex radio-over-fiber system is proposed and experimentally demonstrated based on two Mach-Zehnder modulators and a balanced photodetector (BPD). The cancellation depth is larger than 50 dB and 24 dB for the single frequency and for a wideband signal, respectively. The application of the BPD eliminates the common mode noise and reduces no system flexibility because signal transmission is in one optical fiber by polarization multiplexing technology, instead of two fibers in the traditional self-interference cancellation system with a BPD. In addition, with no electrical delay and attenuation applied, the operational frequency band and cancellation depth are not confined by electronic devices.