We show how careful control of the incident polarization of a light beam close to the Brewster angle gives a giant transverse spatial shift on reflection. This resolves the long-standing puzzle of why such beam shifts transverse to the incident plane (Imbert-Fedorov shifts) tend to be an order of magnitude smaller than the related Goos-Hänchen shifts in the longitudinal direction, which are largest close to critical incidence. We demonstrate that with the proper initial polarization the transverse displacements can be equally large, which we confirm experimentally near Brewster incidence. In contrast to the established understanding, these polarizations are elliptical and angle dependent. We explain the magnitude of the Imbert-Fedorov shift by an analogous change of the symmetry properties for the reflection operators as compared to the Goos-Hänchen shift.