Minimally invasive surgery (MIS) minimizes the surgical incisions that need to be made and hence reduces the physical trauma involved during the surgical process. The ultimate goal is to reduce postoperative pain and blood loss as well as to limit the scarring area and hence accelerate recovery. It is therefore of great interest to both the surgeon and the patient. However, a major problem with MIS is that the field of vision of the surgeon is very narrow. We had previously developed and tested an MIS panoramic endoscope (MISPE) that provides the surgeon with a broader field of view. However, one issue with the MISPE was its low rate of video stitching. Therefore, in this paper, we propose using the region of interest in combination with the downsizing technique to improve the image-stitching performance of the MISPE. Experimental results confirm that, by using the proposed method, the image size can be increased by more than 160%, with the image resolution also improving. For instance, we could achieve performance improvements of 10× (CPU) and 23× (GPU) as compared to that of the original method.