Heavy metal ions, which are over-emitted from industrial production, pose a major threat to the ecological environment and human beings. Among the present detection technologies, achieving rapid and on-site detection of contaminants remains a challenge. Herein, capillaries with three-dimensional (3D) hot spot constructures are fabricated to achieve repaid and ultrasensitive mercury ion (Hg2+) detection in water based on surface-enhanced Raman scattering (SERS). The 4-mercapto pyridine (4-Mpy) serves as the Raman reporter with high selectivity, enabling the detection of Hg2+ by changes in adsorption configuration at the trace level. Under optimized conditions, the SERS response of 4-Mpy for Hg2+ exhibits good linearity, ranging from 1 pM to 0.1 μM in a few minutes, and the detection limit of 0.2 pM is much lower than the maximum Hg2+ concentration of 10 nM allowed in drinking water, as defined by the US Environmental Protection Agency (EPA). Simultaneously, combined with the theoretical simulation and experimental results, the above results indicate that the SERS substrates possess outstanding performances in specificity, recovery rate and stability, which may hold great potential for achieving rapid and on-site environmental pollutant detection using a portable Raman spectrometer.