This study attempted to classify a small amount of electroencephalogram (EEG) data on five states: four tasks involving right index-finger flexion (kinesthetic motor imagery, visual motor imagery, motor execution, and motor observation) and resting with eyes open. We employed a convolutional neural network (CNN) as a classifier and compared the classification accuracies of two types of CNNs: 1) a "single five-class CNN," which classified the aforementioned states with a single CNN and 2) "multiple two-class CNNs," wherein ten CNNs that classify pairs of states were combined. In addition, the classification accuracies were compared between two scenarios: one wherein the EEGs from all 19 scalp probe electrodes (19-channel EEG) were adopted as input data for the CNN, and the other wherein the EEGs of four regions closely related to the motor execution and observation of the index finger (4-channel EEG) were adopted. The classification accuracies of the single five-class CNN with 19- and 4-channel EEGs were 48.2 ± 5.9% and 46.6 ± 6.9%, respectively, and those of the multiple two-class CNNs with 19- and 4-channel EEGs were 52.8 ± 9.7% and 47.5 ± 9.4%, respectively. These results indicate the effectiveness of multiple two-class CNNs that utilize the EEGs of all scalp electrodes as input data for classifying motor imagery, execution, and observation, even in the case of the marginal dataset.