Recent developments propose renewed use of surface-modified nanoparticles (NPs) for enhanced oil recovery (EOR) due to improved stability and reduced porous media retention. The enhanced surface properties render the nanoparticles more suitable compared to bare nanoparticles, for increasing the displacement efficiency of waterflooding. However, the EOR mechanisms using NPs are still not well established. This work investigates the effect of in-situ surface-modified silica nanoparticles (SiO2 NPs) on interfacial tension (IFT) and wettability behavior as a prevailing oil recovery mechanism. For this purpose, the nanoparticles have been synthesized via a one-step sol-gel method using surface-modification agents, including Triton X-100 (non-ionic surfactant) and polyethylene glycol (polymer), and characterized using various techniques. These results exhibit the well-defined spherical particles, particularly in the presence of Triton X-100 (TX-100), with particle diameter between 13 to 27 nm. To this end, SiO2 nanofluids were formed by dispersing nanoparticles (0.05 wt.%, 0.075 wt.%, 0.1 wt.%, and 0.2 wt.%) in 3 wt.% NaCl to study the impact of surface functionalization on the stability of the nanoparticle suspension. The optimal stability conditions were obtained at 0.1 wt.% SiO2 NPs at a basic pH of 10 and 9.5 for TX-100/ SiO2 and PEG/SiO nanofluids, respectively. Finally, the surface-treated SiO2 nanoparticles were found to change the wettability of treated (oil-wet) surface into water-wet by altering the contact angle from 130° to 78° (in case of TX-100/SiO2) measured against glass surface representing carbonate reservoir rock. IFT results also reveal that the surfactant treatment greatly reduced the oil-water IFT by 30%, compared to other applied NPs. These experimental results suggest that the use of surface-modified SiO2 nanoparticles could facilitate the displacement efficiency by reducing IFT and altering the wettability of carbonate reservoir towards water-wet, which is attributed to more homogeneity and better dispersion of surface-treated silica NPs compared to bare-silica NPs.