We present an algorithm to classify polyps in CT colonography images utilizing covariance matrices as object descriptors. Since these descriptors do not lie on a vector space, they cannot simply be fed to traditional machine learning tools such as support vector machines (SVMs) or artificial neural networks (ANNs). To benefit from the simple yet one of the most powerful nonparametric machine learning approach k-nearest neighbor classifier, it suffices to compute the pairwise distances among the covariance descriptors using a distance metric involving their generalized eigenvalues, which also follows from the Lie group structure of positive definite matrices. This approach is fast and discriminates polyps from non-polyps with high accuracy using only a small size descriptor, which consists of 36 unique features per image region extracted from the suspicious regions that we have obtained by combined cellular neural network (CNN) and template matching detection method. These suspicious regions are, in average, 15 x 17 = 255 pixels in our experiments.