Changes in brain state that depend on various visual image stimulations have been investigated recently; however, it is difficult to decode visual image information from brain signal information. Recently, deep learning techniques have been applied to classify brain signals in various experiments, such as motor imagery and steady state visual evoked potential. However, although the deep learning model seems powerful, it is understood poorly, and thus, can be considered a black box. Accordingly, when multi-channel brain signals are trained, which channels include important information is not understood clearly. In this paper, we proposed a channel attention network (CANet) and investigated the way the deep learning network may determine which channels contain more important information that represents brainwaves' characteristics and the way it may visualize that information. Using such spatial channel information, we found that our proposed deep learning architecture outperforms basic approaches (spatial channel information is not considered) to classifying categorized images from visual evoked magnetoencephalographic (MEG) brain signals.