Colorectal cancer is one of the common cancers in the United States. Polyps are one of the major causes of colonic cancer, and early detection of polyps will increase the chance of cancer treatments. In this paper, we propose a novel classification of informative frames based on a convolutional neural network with binarized weights. The proposed CNN is trained with colonoscopy frames along with the labels of the frames as input data. We also used binarized weights and kernels to reduce the size of CNN and make it suitable for implementation in medical hardware. We evaluate our proposed method using Asu Mayo Test clinic database, which contains colonoscopy videos of different patients. Our proposed method reaches a dice score of 71.20% and accuracy of more than 90% using the mentioned dataset.