Deep learning is able to functionally mimic the human brain and thus, it has attracted considerable recent interest. Optics-assisted deep learning is a promising approach to improve forward-propagation speed and reduce the power consumption of electronic-assisted techniques. However, present methods are based on a parallel processing approach that is inherently ineffective in dealing with the serial data signals at the core of information and communication technologies. Here, we propose and demonstrate a sequential optical deep learning concept that is specifically designed to directly process high-speed serial data. By utilizing ultra-short optical pulses as the information carriers, the neurons are distributed at different time slots in a serial pattern, and interconnected to each other through group delay dispersion. A 4-layer serial optical neural network (SONN) was constructed and trained for classification of both analog and digital signals with simulated accuracy rates of over 79.2% with proper individuality variance rates. Furthermore, we performed a proof-of-concept experiment of a pseudo-3-layer SONN to successfully recognize the ASCII codes of English letters at a data rate of 12 gigabits per second. This concept represents a novel one-dimensional realization of artificial neural networks, enabling a direct application of optical deep learning methods to the analysis and processing of serial data signals, while offering a new overall perspective for temporal signal processing.