Homogeneous charge compression ignition (HCCI) is a futuristic automotive engine technology that can significantly improve fuel economy and reduce emissions. HCCI engine operation is constrained by combustion instabilities, such as knock, ringing, misfires, high-variability combustion, and so on, and it becomes important to identify the operating envelope defined by these constraints for use in engine diagnostics and controller design. HCCI combustion is dominated by complex nonlinear dynamics, and a first-principle-based dynamic modeling of the operating envelope becomes intractable. In this paper, a machine learning approach is presented to identify the stable operating envelope of HCCI combustion, by learning directly from the experimental data. Stability is defined using thresholds on combustion features obtained from engine in-cylinder pressure measurements. This paper considers instabilities arising from engine misfire and high-variability combustion. A gasoline HCCI engine is used for generating stable and unstable data observations. Owing to an imbalance in class proportions in the data set, the models are developed both based on resampling the data set (by undersampling and oversampling) and based on a cost-sensitive learning method (by overweighting the minority class relative to the majority class observations). Support vector machines (SVMs) and recently developed extreme learning machines (ELM) are utilized for developing dynamic classifiers. The results compared against linear classification methods show that cost-sensitive nonlinear ELM and SVM classification algorithms are well suited for the problem. However, the SVM envelope model requires about 80% more parameters for an accuracy improvement of 3% compared with the ELM envelope model indicating that ELM models may be computationally suitable for the engine application. The proposed modeling approach shows that HCCI engine misfires and high-variability combustion can be predicted ahead of time, given the present values of available sensor measurements, making the models suitable for engine diagnostics and control applications.