Spherical implosions in inertial confinement fusion are inherently sensitive to perturbations that may arise from experimental constraints and errors. Control and mitigation of low-mode (long wavelength) perturbations is a key milestone to improving implosion performances. We present the first 3D radiation-hydrodynamic simulations of directly driven inertial confinement fusion implosions with an inline package for polarized crossed-beam energy transfer. Simulations match bang times, yields (separately accounting for laser-induced high modes and fuel age), hot spot flow velocities and direction, for which polarized crossed-beam energy transfer contributes to the systematic flow orientation evident in the OMEGA implosion database. Current levels of beam mispointing, imbalance, target offset, and asymmetry from polarized crossed-beam energy transfer degrade yields by more than 40%. The effectiveness of two mitigation strategies for low modes is explored.