SERS (Surface Enhanced Raman Spectroscopy) can realize fingerprint recognition of molecular samples with high detection accuracy and sensitivity. However, rapid and convenient measurement of the Raman spectra of analytes for a point-of-care test (POCT) has put forward a high demand for portable Raman spectrometers, as well as reliable SERS substrates. Hereby, we first utilize a smartphone as a miniaturized Raman spectral analyzer, which has the revolutionary advantages of a friendly human-machine interface, fast measurement time, and good sensitivity. Meanwhile, a paper-based SERS chip was prepared based on commonly used filter paper and silver nanoparticles (AgNP), which was successfully used to detect low concentrations of typical SERS analyte model molecules including rhodamine 6G and crystal violet. The current method of smartphone-based SERS spectroscopy as a POCT device will greatly promote the application of Raman technology in a variety of scenarios, such as safety inspections, pesticide residue detection, water pollution monitoring, and so on. Coupled with paper-based SERS chips with advantages of facile preparation, low cost and good reliability, the current work proves to have a great potential for industrial production and for meeting the vast marketing demand of Raman based POCT technology.