We propose a marker-based geometric framework for the high-frequency absolute 3D pose estimation of a binocular camera system by using the data captured during the exposure of a single rolling shutter scanline. In contrast to existing approaches enforcing temporal or motion models among scanlines (e.g. linear motion, constant velocity or small motion assumptions), we strive to determine the pose from instantaneous binocular capture (i.e. without using data from previous scanlines) and achieve drift-free pose estimation. We leverage the projective invariants of a novel rigid planar pattern, to both define a geometric reference as well as to determine 2D-3D correspondences from raw edge detection measurements from individual scanlines. Moreover, to tackle the ensuing multi-view estimation problem, achieve real-time operation, and minimize latency, we develop a pair of custom solvers leveraging our geometric setup. To mitigate sensitivity to noise, we propose a geometrically consistent measurement refinement mechanism. We verify the quality of our solvers by comparing with state of the art general solvers for absolute pose estimation of generalized cameras. Finally, we demonstrate the effectiveness of our proposed approach with an FPGA-based implementation which achieves a localization throughput of 129.6 KHz with a 1.5 μs latency.