In stereo-matching techniques for three-dimensional (3D) vision, illumination change is a major problem that degrades matching accuracy. When large intensity differences are observed between a pair of stereos, it is difficult to find the similarity in the matching process. In addition, inaccurately estimated disparities are obtained in textureless regions, since there are no distinguishable features in the region. To solve these problems, this paper presents a robust stereo-matching method using illuminant-invariant cost volume and confidence-based disparity refinement. In the step of matching a stereo pair, the proposed method combines two cost volumes using an invariant image and Weber local descriptor (WLD), which was originally motivated by human visual characteristics. The invariant image used in the matching step is insensitive to sudden brightness changes by shadow or light sources, and WLD reflects structural features of the invariant image with consideration of a gradual illumination change. After aggregating the cost using a guided filter, we refine the initially estimated disparity map based on the confidence map computed by the combined cost volume. Experimental results verify that the matching computation of the proposed method improves the accuracy of the disparity map under a radiometrically dynamic environment. Since the proposed disparity refinement method can also reduce the error of the initial disparity map in textureless areas, it can be applied to various 3D vision systems such as industrial robots and autonomous vehicles.