Ellipsometry is one of the standard methods for observation of glass transition in thin polymer films. This work proposes that sensitivity of the method to surface morphology can complicate manifestation of the transition in a few nm thick samples. Two possible mechanisms of free surface roughening in the vicinity of glass transition are discussed: roughening due to lateral heterogeneity and roughening associated with thermal capillary waves. Both mechanisms imply an onset of surface roughness in the glass transition temperature range, which affects the experimental data in a way that shifts apparent glass transition temperature. Effective medium approximation models are used to introduce surface roughness into optical calculations. The results of the optical modeling for a 5 nm thick polystyrene film on silicon are presented.