The human visual system efficiently extracts local elements from cluttered backgrounds and integrates these elements into meaningful contour perception. This process is a critical step before object recognition, in which contours often play an important role in defining the shapes and borders of the to-be-recognized objects. However, the neural mechanism of the contour integration is still under debate. The investigation of the neural mechanism underlying contour integration could deepen our understanding of perceptual grouping in the human visual system and advance the development of the algorithms for image grouping and segmentation in computer vision. Here, we review two theoretical frameworks that were proposed over the past decades. The first framework is based on hardwired horizontal connection in primary visual cortex, while the second one emphasizes the role of recurrent connections within intra- and inter-areas. At the end of review, we also raise the unsolved issues that need to be addressed in future studies.