Improving the photo-induced charge transfer (PICT) efficiency by adjusting the energy levels difference between adsorbed probe molecules and substrate materials is a key factor for boosting the surface enhanced Raman scattering (SERS) based on the chemical mechanism (CM). Herein, a new route to improve the SERS activity of two-dimensional (2D) selenium and tin compounds (SnSex, 1 ≤ x ≤ 2) by the hybrid phase materials is researched. The physical properties and the energy band structure of SnSex were analyzed. The enhanced SERS activity of 2D SnSex can be attribute to the coupling of the PICT resonance caused by the defect energy levels induced by Se vacancy and the molecular resonance Raman scattering (RRS). This established a relationship between the physical properties and SERS activity of 2D layered materials. The resonance probe molecule, rhodamine (R6G), which is used to detect the SERS performance of SnSex nanosheets. The enhancement factor (EF) of R6G on the optimized SnSe1.35 nanosheets can be as high as 2.6 × 106, with a detection limit of 10-10 M. The SERS result of the environmental pollution, thiram, shows that the SnSex nanosheets have a practical application in trace SERS detection, without the participation of metal particles. These results demonstrate that, through hybrid phase materials, the SERS sensitivity of 2D layered nanomaterials can be improved. It provides a kind of foreground non-metal SERS substrate in monitoring or detecting and provide a deep insight into the chemical SERS mechanism based on 2D layered materials.