A thermal radiative component is likely to accompany the first stages of the prompt emission of gamma-ray bursts (GRBs) and X-ray flashes. We analyse the effect of such a component on the observable spectrum, assuming that the observable effects are due to a dissipation process occurring below or near the thermal photosphere. For comparable energy densities in the thermal and leptonic components, the dominant emission mechanism is Compton scattering. This leads to a nearly flat energy spectrum (nuFnu proportional, 0) above the thermal peak at approximately 10-100 keV and below 10-100 MeV, for a wide range of optical depths 0.03 less, similar tau less, similar 100, regardless of the details of the dissipation mechanism or the strength of the magnetic field. For higher values of the optical depth, a Wien peak is formed at 100 keV to 1 MeV. In particular, these results are applicable to the internal shock model of GRBs, as well as to slow dissipation models, e.g. as might be expected from reconnection, if the dissipation occurs at a sub-photospheric radii. We conclude that dissipation near the thermal photosphere can naturally explain (i) clustering of the peak energy at sub-MeV energies at early times, (ii) steep slopes observed at low energies, and (iii) a flat spectrum above 10 keV at late times. Our model thus provides an alternative scenario to the optically thin synchrotron-synchrotron self-Compton model.