We consider an invariant random matrix ensemble where the standard Gaussian potential is distorted by an additional single pole of arbitrary fixed order. Potentials with first- and second-order poles have been considered previously and found applications in quantum chaos and number theory. Here we present an application to mean-field glassy systems. We derive and solve the loop equation in the planar limit for the corresponding class of potentials. We find that the resulting mean or macroscopic spectral density is generally supported on two disconnected intervals lying on the two sides of the repulsive pole, whose edge points can be completely determined imposing the additional constraint of traceless matrices on average. For an unbounded potential with an attractive pole, we also find a possible one-cut solution for certain values of the couplings, which is ruled out when the traceless condition is imposed. Motivated by the calculation of the distribution of the spin-glass susceptibility in the Sherrington-Kirkpatrick spin-glass model, we consider in detail a second-order pole for a zero-trace model and provide the most explicit solution in this case. In the limit of a vanishing pole, we recover the standard semicircle. Working in the planar limit, our results apply to matrices with orthogonal, unitary, and symplectic invariance. Numerical simulations and an independent analytical Coulomb fluid calculation for symmetric potentials provide an excellent confirmation of our results.