Using resonant x-ray diffraction, we observe an easy c-axis collinear antiferromagnetic structure for the bilayer Sr3Ir2O7, a significant contrast to the single layer Sr2IrO4 with in-plane canted moments. Based on a microscopic model Hamiltonian, we show that the observed spin-flop transition as a function of number of IrO2 layers is due to strong competition among intra- and interlayer bond-directional pseudodipolar interactions of the spin-orbit entangled J(eff)=1/2 moments. With this we unravel the origin of anisotropic exchange interactions in a Mott insulator in the strong spin-orbit coupling regime, which holds the key to the various types of unconventional magnetism proposed in 5d transition metal oxides.