Based on the polarization-sensitive absorption of graphene under conditions of total internal reflection, a novel optical sensor combining graphene and a microfluidic structure was constructed to achieve the sensitive real-time monitoring of refractive indexes. The atomic thickness and strong broadband absorption of graphene cause it to exhibit very different reflectivity for transverse electric and transverse magnetic modes in the context of a total internal reflection structure, which is sensitive to the media in contact with the graphene. A graphene refractive index sensor can quickly and sensitively monitor changes in the local refractive index with a fast response time and broad dynamic range. These results indicate that graphene, used in a simple and efficient total internal reflection structure and combined with microfluidic techniques, is an ideal material for fabricating refractive index sensors and biosensor devices, which are in high demand.