Self-interference (SI) due to signal leakage from a local transmitter is an issue in an in-band full-duplex (IBFD) transmission system, which would cause severe distortions to a receiving signal of interest (SOI). By superimposing a local reference signal with the same amplitude and opposite phase, the SI signal can be fully canceled. However, as the manipulation of the reference signal is usually operated manually, it is difficult to ensure a high speed and high accurate cancellation. To overcome this problem, a real-time adaptive optical SI cancellation (RTA-OSIC) scheme using a SARSA(λ) reinforcement learning (RL) algorithm is proposed and experimentally demonstrated. The proposed RTA-OSIC scheme can automatically adjust the amplitude and phase of a reference signal by adjusting a variable optical attenuator (VOA) and a variable optical delay line (VODL) achieved through an adaptive feedback signal, which is generated by evaluating the quality of the received SOI. To verify the feasibility of the proposed scheme, a 5 GHz 16QAM OFDM IBFD transmission experiment is demonstrated. By using the proposed RTA-OSIC scheme, for an SOI at three different bandwidths of 200, 400, and 800 MHz, the signal can be adaptively and correctly recovered within 8 time periods (TPs), which is the required time of a single adaptive control step. The cancellation depth for the SOI with a bandwidth of 800 MHz is 20.18 dB. The short- and long-term stability of the proposed RTA-OSIC scheme is also evaluated. The experimental results indicate that the proposed approach could be a promising solution for real-time adaptive SI cancellation in future IBFD transmission systems.