In this paper, we propose a progressive reliable points growing matching scheme to estimate the depth from the speckle projection image. First a self-adapting binarization is introduced to reduce the influence of inconsistent intensity. Then we apply local window-based correlation matching to get the initial disparity map. After the initialization, we formulate a progressive updating scheme to update the disparity estimation. There are two main steps in each round of updation. At first new reliable points are progressively selected based on three aspects of criterion including matching degree, confidence, and left-right consistency; then prediction-based growing matching is adopted to recalculate the disparity map from the reliable points. Finally, the more accurate depth map can be obtained by subpixel interpolation and transformation. The experimental results well demonstrate the effectiveness and low computational cost of our scheme.