Direct detection system is expected to possess the phase and polarization diversity in order to achieve high spectral efficiency and fiber impairment compensation such as chromatic dispersion and polarization rotation. In this Letter, we theoretically extend the concept of the proposed Jones-space field recovery (JSFR) to include a dynamic polarization rotation matrix and experimentally demonstrate the rapid polarization state tracking ability of the JSFR receiver based on a 3 × 3 optical coupler. Under a rotation of the state of polarization at a rate of 1 Mrad/s, we successfully transmit 59-GBd dual-polarization 16-ary quadrature-amplitude-modulation signals over an 80-km standard single-mode fiber based on a decision-directed least mean square (DD-LMS) or a recursive least square (DD-RLS), with a bit-error rate below the 14% hard-decision forward error correction threshold of 1 × 10-2. The experimental results indicate that the legacy polarization tracking algorithms designed for coherent optical communication are also applicable for this direct detection scheme. To our best knowledge, this work demonstrates the first polarization rotation-tolerant direct detection system with phase and polarization diversity, providing a low-cost and high-speed solution for short-reach communications.