In this paper, we treat the problem of reducing the false positives (FP) in the automatic detection of colorectal polyps at Computer Aided Detection in Computed Tomography Colonography (CAD-CTC) as a shape-filtering task. From the extracted candidate surface, we obtain a reliable shape distribution function and analyse it in the Fourier domain and use the resulting spectral data to classify the candidate surface as belonging to a polyp or a non-polyp class. The developed shape filtering scheme is computationally efficient (takes approximately 2 seconds per dataset to detect the polyps from the colonic surface) and offers robust polyp detection with an overall false positive rate of 5.44 per dataset at a sensitivity of 100% for polyps greater than 10 mm when it was applied to standard and low dose CT data.