The visible light communication channel has time-varying characteristics and is difficult to predict. This paper proposes an equalization algorithm based on the structure of a convolutional neural network (CNN), combining time series feature length and long short-term memory (LSTM), and adding a residual structure. It can be seen that the equalization coefficient vector of the optical channel is a time series, which can reflect the noise characteristics of the channel and has storage characteristics. The equalizer algorithm can accurately learn the complex channel characteristics and calculate the compensation coefficient according to the channel characteristics. It can also restore the original transmission signal. At the same time, this paper also examines the compensation method of the receiver in the mobile state. The long-term memory parameters of LSTM are used to represent the sequence causality in the memory channel, and CNN and residual structure are used to refine the results and improve the accuracy of the reconstruction. The simulation results show that the algorithm can effectively eliminate the influence of the fading characteristics of the visible optical channel, improve the bit error rate performance of system transmission, solve the overall problem of channel corruption, and precisely restore the original transmission signal with fast convergence speed. In addition, this method can achieve a better balance between performance and complexity compared to the traditional contention balancing method, which proves the potential and effectiveness of the proposed channel balancing method.