Pap test involves searching of morphological changes in cervical squamous epithelial cells by pathologists or cytotechnologists to identify potential cancerous cells in the cervix. Nuclear membrane irregularity is one of the morphological changes of malignancy. This paper proposes two novel techniques for the evaluation of nuclear membrane irregularity. The first technique, namely, penalty-driven smoothing analysis, introduces different penalty values for nuclear membrane contour with different degrees of irregularity. The second technique, which can be subdivided into mean- or median-type residual-based analysis, computes the number of points of nuclear membrane contour that deviates from the mean or median of the nuclear membrane contour. Performance of the proposed techniques was compared to three state-of-the-art techniques, namely, radial asymmetric, shape factor, and rim difference. Friedman and post hoc tests using Holm, Shaffer, and Bergmann procedures returned significant differences for all the three classes, i.e., negative for intraepithelial lesion or malignancy (NILM) versus low grade squamous intraepithelial lesion (LSIL), NILM versus high grade squamous intraepithelial lesion (HSIL), and LSIL versus HSIL when the span value equaled 3 was employed with linear penalty function. When span values equaled 5, 7, and 9, NILM versus LSIL and HSIL showed significant differences regardless of the penalty functions. In addition, the results of penalty-driven smoothing analysis were comparable with those of other state-of-the-art techniques. Residual-based analysis returned significant differences for the comparison among the three diagnostic classes. Findings of this study proved the significance of nuclear membrane irregularity as one of the features to differentiate the different diagnostic classes of cervical squamous epithelial cells.