Sleep disorders have become more common due to the modern lifestyle and stress. The most severe case of sleep disorders called apnea is characterized by a complete breaking block, leading to awakening and subsequent sleep disturbances. The automatic detection of sleep arousals is still challenging. In this paper, a novel method is presented to detect non-apnea sources of arousals during sleep using Polysomnography(PSG) recordings. After the preprocessing, a sequence-to-sequence deep neural network (DNNs) consisting of a series of Bidirectional long short-term memory (Bi-LSTM) layer, and fully connected layers were trained to classify samples in the segments. Initially, three different models were prepared for different datasets. Finally, obtaining the classification result through an ensemble model consisting of the three trained models. The result shows that the area under the receiver precision-recall curve (AUPRC) is 0.59 for the test dataset exceeding the performance of the classifiers in the existing literature.Clinical relevance- Analyzing Polysomnographic recordings is a time consuming a critical process yet to identify sleep disorders. These recordings span several hours and contain different data streams that include EEG, EMG, etc. This paper proposes a system that can automatically detect respiratory effort-related arousals using a deep neural network from Polysomnographic Recordings. By automating this process with a machine learning-based solution that can eliminate the manual process while facilitating further improvements of the system with future data.