The basis set limit Møller-Plesset second-order equilibrium bond lengths of He2, Be2, and Ne2, accurate to 0.01a0, are computed to be 5.785a0, 5.11a0, and 6.05a0. The corresponding binding energies are 22.4+/-0.1, 2180+/-20, and 86+/-2 muE(h), respectively. An accuracy of 95% in the binding energy requires an aug-cc-pV6Z basis or larger for conventional Møller-Plesset theory. This accuracy is obtained using an aug-cc-pV5Z basis if geminal basis functions with a linear correlation factor are included and with an aug-cc-pVQZ basis if the linear correlation factor is replaced by exp(-gammar(12)) with gamma=1. The correlation factor r(12) exp(-gammar(12)) does not perform as well, describing the atom more efficiently than the dimer. The geminal functions supplement the orbital basis in the description of both the short-range correlation, at electron coalescence, and the long-range dispersion correlation and the values of gamma that give the best binding energies are smaller than those that are optimum for the atom or the dimer. It is important to sufficiently reduce the error due to the resolution of the identity approximation for the three- and four-electron integrals and we recommend the complementary auxiliary basis set method. The effect of both orbital and geminal basis set superposition error must be considered to obtain accurate binding energies with small orbital basis sets. In this respect, we recommend using exp(-gammar(12)) with localized orbitals and the original orbital-variant formalism.