We propose and demonstrate the asymmetric direct detection (ADD) of polarization division multiplexed single-sideband (PDM-SSB) signals with orthogonal offset carriers. ADD exploits the photocurrent difference to eliminate the Y-Pol interference in the X-Pol, and the X-Pol signal intensity to eliminate the X-Pol interference in the Y-Pol without resorting to iterative algorithms. This enables not only low-complexity signal linearization but also a simplified receiver front-end composed of a single optical filter, two single-ended photodiodes and two analog-to-digital converters (ADC). In the experiment, we first perform a parametric study of the proposed scheme at 40 Gbaud in the back-to-back configuration (B2B) to evaluate the performance impact of different system parameters including the carrier to signal power ratio (CSPR), the matched filter roll-off, and the filter guard band. Next, we demonstrate the transmission of 416 Gbit/s PDM 16-QAM signal over 80 km single-mode fiber (SMF) below the soft-decision forward error correction (SD-FEC) threshold of 2×10-2. We also numerically study the effectiveness of a 2×2 multiple-input-multiple-output MIMO equalizer in alleviating the inter-polarization linear crosstalk resulting from the non-orthogonal PDM-SSB signals due to polarization-dependent loss (PDL), which is not negligible for potential on-chip implementation of ADD.