Sign-changing interactions constitute a crucial ingredient in the creation of frustrated many-body systems such as spin glasses. We present here the demonstration of a photon-mediated sign-changing interaction between Bose-Einstein-condensed atoms in a confocal cavity. The interaction between two atoms is of an unusual, nonlocal form proportional to the cosine of the inner product of the atoms' position vectors. This interaction arises from the differing Gouy phase shifts of the cavity's degenerate modes. The interaction drives a nonequilibrium Dicke-type phase transition in the system leading to atomic checkerboard density-wave order. Because of the Gouy phase anomalies, the checkerboard pattern can assume either a sinelike or cosinelike character. This state is detected via the holographic imaging of the cavity's superradiant emission. Together with a companion paper [Y. Guo, V. D. Vaidya, R. M. Kroeze, R. A. Lunney, B. L. Lev, and J. Keeling, Emergent and broken symmetries of atomic self-organization arising from Gouy phases in multimode cavity QED, Phys. Rev. A 99, 053818 (2019)PLRAAN2469-992610.1103/PhysRevA.99.053818], we explore this interaction's influence on superradiant phase transitions in multimode cavities. Employing this interaction in cavity QED spin systems may enable the creation of artificial spin glasses and quantum neural networks.