Entomopathogenic nematodes including Steinernema spp. play an increasingly important role as biological alternatives to chemical pesticides. The infective juveniles of these worms use nictation - a behavior in which animals stand on their tails - as a host-seeking strategy. The developmentally-equivalent dauer larvae of the free-living nematode Caenorhabditis elegans also nictate, but as a means of phoresy or "hitching a ride" to a new food source. Advanced genetic and experimental tools have been developed for C. elegans , but time-consuming manual scoring of nictation slows efforts to understand this behavior, and the textured substrates required for nictation can frustrate traditional machine vision segmentation algorithms. Here we present a Mask R-CNN-based tracker capable of segmenting C. elegans dauers and S. carpocapsae infective juveniles on a textured background suitable for nictation, and a machine learning pipeline that scores nictation behavior. We use our system to show that the nictation propensity of C. elegans from high-density liquid cultures largely mirrors their development into dauers, and to quantify nictation in S. carpocapsae infective juveniles in the presence of a potential host. This system is an improvement upon existing intensity-based tracking algorithms and human scoring which can facilitate large-scale studies of nictation and potentially other nematode behaviors.