Second-order perturbation theory based on the Kohn-Sham Hamiltonian leads to an implicit density functional for the correlation energy E(c) (MP2), which is explicitly dependent on both occupied and unoccupied Kohn-Sham single-particle orbitals and energies. The corresponding correlation potential v(c) (MP2), which has to be evaluated by the optimized potential method, was found to be divergent in the asymptotic region of atoms, if positive-energy continuum states are included in the calculation [Facco Bonetti et al., Phys. Rev. Lett. 86, 2241 (2001)]. On the other hand, Niquet et al., [J. Chem. Phys. 118, 9504 (2003)] showed that v(c) (MP2) has the same asymptotic -alpha(2r(4)) behavior as the exact correlation potential, if the system under study has a discrete spectrum only. In this work we study v(c) (MP2) for atoms in a spherical cavity within a basis-set-free finite differences approach, ensuring a completely discrete spectrum by requiring hard-wall boundary conditions at the cavity radius. Choosing this radius sufficiently large, one can devise a numerical continuation procedure which allows to normalize v(c) (MP2) consistent with the standard choice v(c)(r-->infinity)=0 for free atoms, without modifying the potential in the chemically relevant region. An important prerequisite for the success of this scheme is the inclusion of very high-energy virtual states. Using this technique, we have calculated v(c) (MP2) for all closed-shell and spherical open-shell atoms up to argon. One finds that v(c) (MP2) reproduces the shell structure of the exact correlation potential very well but consistently overestimates the corresponding shell oscillations. In the case of spin-polarized atoms one observes a strong interrelation between the correlation potentials of the two spin channels, which is completely absent for standard density functionals. However, our results also demonstrate that E(c) (MP2) can only serve as a first step towards the construction of a suitable implicit correlation functional: The fundamental variational instability of this functional is recovered for beryllium, for which a breakdown of the self-consistent Kohn-Sham iteration is observed. Moreover, even for those atoms for which the self-consistent iteration is stable, the results indicate that the inclusion of v(c) (MP2) in the total Kohn-Sham potential does not lead to an improvement compared to the complete neglect of the correlation potential.