Time-domain diffuse optical tomography can efficiently reconstruct both absorption and reduced scattering coefficients but is heavily limited by the ill-posedness in its inverse problem and low spatial resolution. To deal with these adversities, the truncated singular value decomposition (TSVD)-based whole-weighting-matrix inversion scheme can be a particularly suitable implementation. Unfortunately, TSVD is subject to a storage challenge for three-dimensional imaging of a bulk region, such as breast. In this paper, a multi-scale mesh strategy based on computed tomography (CT) anatomical geometry is adopted to solve the storage challenge, where a fine mesh is used in forward calculation to ensure accuracy, and a coarse mesh in the inversion process to enable TSVD-based inversion of the whole-weighting matrix. We validate the proposed strategy using simulated data for a single lesion model from clinical positron emission tomography images of a breast cancer patient, and further, for a complex model that is constructed by setting dual lesions at different separations in the CT breast geometry.