Inspired by the non-regular framework studied in Laber and Murphy (2011), we propose a family of adaptive classifiers. We discuss briefly their asymptotic properties and show that under the non-regular framework these classifiers have an "oracle property," and consequently have smaller asymptotic variance and smaller asymptotic test error variance than those of the original classifier. We also show that confidence intervals for the test error of the adaptive classifiers, based on either normal approximation or centered percentile bootstrap, are consistent.