The Markov random field (MRF) for stereo matching can be solved using belief propagation (BP). However, the solution space grows significantly with the introduction of high-resolution stereo images and 3D plane labels, making the traditional BP algorithms impractical in inference time and convergence. We present an accurate and efficient hierarchical BP framework using the representation of the image segmentation pyramid (ISP). The pixel-level MRF can be solved by a top-down inference on the ISP. We design a hierarchy of MRF networks using the graph of superpixels at each ISP level. From the highest/image to the lowest/pixel level, the MRF models can be efficiently inferred with constant global guidance using the optimal labels of the previous level. The large texture-less regions can be handled effectively by the MRF model on a high level. The advanced 3D continuous labels and a novel support-points regularization are integrated into our framework for stereo matching. We provide a data-level parallelism implementation which is orders of magnitude faster than the best graph cuts (GC) algorithm. The proposed framework, HBP-ISP, outperforms the best GC algorithm on the Middlebury stereo matching benchmark.