Plant-parasitic nematodes (PPNs) cause serious harm to agricultural production. Bacillus firmus shows excellent control of PPNs and has been produced as a commercial nematicide. However, its nematicidal factors and mechanisms are still unknown. In this study, we showed that B. firmus strain DS-1 has high toxicity against Meloidogyne incognita and soybean cyst nematode. We sequenced the whole genome of DS-1 and identified multiple potential virulence factors. We then focused on a peptidase S8 superfamily protein called Sep1 and demonstrated that it had toxicity against the nematodes Caenorhabditis elegans and M. incognita. The Sep1 protein exhibited serine protease activity and degraded the intestinal tissues of nematodes. Thus, the Sep1 protease of B. firmus is a novel biocontrol factor with activity against a root-knot nematode. We then used C. elegans as a model to elucidate the nematicidal mechanism of Sep1, and the results showed that Sep1 could degrade multiple intestinal and cuticle-associated proteins and destroyed host physical barriers. The knowledge gained in our study will lead to a better understanding of the mechanisms of B. firmus against PPNs and will aid in the development of novel bio-agents with increased efficacy for controlling PPNs.