The quest for providing tissue characterization and functional mapping during minimally invasive surgery (MIS) has motivated the development of new surgical tools that extend the current functional capabilities of MIS. Miniaturized optical probes can be inserted into the instrument channel of standard endoscopes to reveal tissue cellular and subcellular microstructures, allowing excision-free optical biopsy. One of the limitations of such a point based imaging and tissue characterization technique is the difficulty of tracking probed sites in vivo. This prohibits large area surveillance and integrated functional mapping. The purpose of this paper is to present an image-based tracking framework by combining a semi model-based instrument tracking method with vision-based simultaneous localization and mapping. This allows the mapping of all spatio-temporally tracked biopsy sites, which can then be re-projected back onto the endoscopic video to provide a live augmented view in vivo, thus facilitating re-targeting and serial examination of potential lesions. The proposed method has been validated on phantom data with known ground truth and the accuracy derived demonstrates the strength and clinical value of the technique. The method facilitates a move from the current point based optical biopsy towards large area multi-scale image integration in a routine clinical environment.