The structure of AgN clusters (N=1-4, 6, 8, 10), both in the gas phase and grown on the MgO(1 0 0) surface containing Fs-defects, has been investigated by a density functional basin-hopping (DF-BH) approach. In analogy with what observed in the case of gold clusters, it is found that the presence of the defect implies a double frustration and a cylindrical invariance of the metal-surface interaction, causing small Ag clusters growing around the Fs defect to be highly fluxional. Nevertheless, two different structural crossovers are found to be induced by the metal-defect interaction for the adsorbed clusters such that: 1) planar structures prevail for N<or=4 (as in the gas phase); 2) noncrystalline (fivefold symmetric) structures, which are the lowest energy ones in the gas phase for medium sized AgN clusters (N>or=7), prevail for N=6 and N=8; 3) distorted face-centered cubic (fcc) structures grown pseudomorphically on the defected surface prevail for N=10. The transition from fivefold to fcc motifs is rationalized in terms of the double-frustration effect, which increases the bond strain of the noncrystalline structures. Detrapping energies from the defect were also calculated; the lowest energy pathway corresponds to the detachment of a dimer.