A novel method of using stereoscopic video images to synthesize the computer-generated hologram (CGH) patterns of a real 3D object is proposed. Stereoscopic video images of a real 3D object are captured by a 3D camera system. Disparity maps between the captured stereo image pairs are estimated and from these estimated maps the depth data for each pixel of the object can be extracted on a frame basis. By using these depth data and original color images, hologram patterns of a real object can be computationally generated. In experiments, stereoscopic video images of a real 3D object, a wooden rhinoceros doll, are captured by using the Wasol 3D adapter system and its depth data are extracted from them. Then, CGH patterns of 1280 pixels x 1024 pixels are generated with these depth-annotated images of the wooden rhinoceros doll, and the CGH patterns are experimentally displayed via a holographic display system.