While much is known about repulsive quantum impurity models, significantly less attention has been devoted to their attractive counterparts. This motivated us to study the attractive SU(N) Anderson impurity model. While for the repulsive case the phase diagram features mild N dependence and the ground state is always a Fermi liquid, in the attractive case a Kosterlitz-Thouless charge localization phase transition is revealed for N>2. Beyond a critical value of attractive interaction, an abrupt jump appears in the number of particles at the impurity site, and a singular Fermi liquid state emerges, where the scattering of quasiparticles is found to exhibit power law behavior with fractional power. The capacity diverges exponentially at the quantum critical point, signaling the Kosterlitz-Thouless transition.