The refractive index and extinction coefficient of chemical vapour deposition grown graphene are determined by ellipsometry analysis. Graphene films were grown on copper substrates and transferred as both monolayers and bilayers onto SiO2/Si substrates by using standard manufacturing procedures. The chemical nature and thickness of residual debris formed after the transfer process were elucidated using photoelectron spectroscopy. The real layered structure so deduced has been used instead of the nominal one as the input in the ellipsometry analysis of monolayer and bilayer graphene, transferred onto both native and thermal silicon oxide. The effect of these contamination layers on the optical properties of the stacked structure is noticeable both in the visible and the ultraviolet spectral regions, thus masking the graphene optical response. Finally, the use of heat treatment under a nitrogen atmosphere of the graphene-based stacked structures, as a method to reduce the water content of the sample, and its effect on the optical response of both graphene and the residual debris layer are presented. The Lorentz-Drude model proposed for the optical response of graphene fits fairly well the experimental ellipsometric data for all the analysed graphene-based stacked structures.