Attosecond pump-probe experiments routinely utilize extreme ultraviolet (XUV) and IR fields, with relative phase being the variable parameter. However, the Gouy phase slip between the focused IR and XUV pulses inevitably leads to a certain amount of phase averaging and loss of accuracy. By using ion imaging, we establish a one-to-one mapping between the local phase slip and the spatial coordinates of the focal volume, thus performing in situ characterization of the Gouy phase of a complex beam and its role in ionization of He and Xe. We demonstrate that spatially discriminated ion imaging enhances the contrast of a phase-dependent XUV+IR ionization signal. We utilize our technique to unmask a weak ionization asymmetry, thus opening pathways for further high-precision attosecond studies.