The accuracy of strain measurement using a common optical extensometer with two-dimensional (2D) digital image correlation (DIC) is not sufficient for experimental applications due to the effect of out-of-plane motion. Although three-dimensional (3D) DIC can measure all three components of displacement without introducing in-plane displacement errors, 3D-DIC requires the stringent synchronization between two digital cameras and requires complicated system calibration of binocular stereovision, which makes the measurement rather inconvenient. To solve the problems described above, this paper proposes a self-calibration single-lens 3D video extensometer for non-contact, non-destructive and high-accuracy strain measurement. In the established video extensometer, a single-lens 3D imaging system with a prism and two mirrors is constructed to acquire stereo images of the test sample surface, so the problems of synchronization and out-of-plane displacement can be solved easily. Moreover, a speckle-based self-calibration method which calibrates the single-lens stereo system using the reference speckle image of the specimen instead of the calibration targets is proposed, which will make the system more convenient to be used without complicated calibration. Furthermore, an efficient and robust inverse compositional Gauss-Newton algorithm combined with a robust stereo matching stage is employed to achieve high-accuracy and real-time subset-based stereo matching. Tensile tests of an Al-alloy specimen were performed to demonstrate the feasibility and effectiveness of the proposed self-calibration single-lens 3D video extensometer.