Surface enhanced Raman scattering (SERS) fiber sensors have shown great potential in sensitive biosensing and medical diagnostics. However, current SERS fiber probes are most commonly based on stiff silica fibers, which, unfortunately, are not mechanically compliant with soft biological tissues. In addition, the poor biocompatibility of silica fibers sets another barrier that hinders their development for biomedical applications. Here, we present, to the best of our knowledge, the first demonstration of soft-polymer-optical-fiber-based SERS (SPOF-SERS) probes with physio-mechanical properties suitable for implantation, and demonstrate their potential applications for in situ detection of bioanalysts. The SPOFs are made from porous hydrogel materials that are soft, elastic, and biocompatible. The three-dimensional porous structures of the hydrogels enable high loading of metal nanoparticles to provide a large amount of SERS "hot spots" for high sensitivity. We tested the SPOF-SERS sensor for detection and discrimination of rhodamine 6G and 4-mercaptopyridine in situ with detection limits of 10-7 M and 10-8 M, respectively. We also demonstrated the capability of SPOF-SERS probes in multiplexing detection. The soft, biocompatible, and highly sensitive SERS probe is promising for bioanalytical and implantable biomedical applications.