The significant bottlenecks in determining bacterial species are much more time-consuming and the biology specialist's long-term experience requirements. Specifically, it takes more than half a day to cultivate a bacterium, and then a skilled microbiologist and a costly specialized machine are utilized to analyze the genes and classify the bacterium according to its nucleotide sequence. To overcome these issues as well as get higher recognition accuracy, we proposed applying convolutional neural networks (CNNs) architectures to automatically classify bacterial species based on some key characteristics of bacterial colonies. Our experiment confirmed that the classification of three bacterial colonies could be performed with the highest accuracy (97.19%) using a training set of 5000 augmented images derived from the 40 original photos taken in the Hanoi Medical University laboratory in Vietnam.