High energy X-ray phase contrast tomography is tremendously beneficial to the study of thick and dense materials with poor attenuation contrast. Recently, the X-ray speckle-based imaging technique has attracted widespread interest because multimodal contrast images can now be retrieved simultaneously using an inexpensive wavefront modulator and a less stringent experimental setup. However, it is time-consuming to perform high resolution phase tomography with the conventional step-scan mode because the accumulated time overhead severely limits the speed of data acquisition for each projection. Although phase information can be extracted from a single speckle image, the spatial resolution is deteriorated due to the use of a large correlation window to track the speckle displacement. Here we report a fast data acquisition strategy utilising a fly-scan mode for near field X-ray speckle-based phase tomography. Compared to the existing step-scan scheme, the data acquisition time can be significantly reduced by more than one order of magnitude without compromising spatial resolution. Furthermore, we have extended the proposed speckle-based fly-scan phase tomography into the previously challenging high X-ray energy region (120 keV). This development opens up opportunities for a wide range of applications where exposure time and radiation dose are critical.