The interaction of bitumen colloidal (a form of heavy oil) with inorganic solids, for example, mineral aggregates, in both air and water environments is ubiquitous in nature and engineering. However, our knowledge of the underlying physical mechanism of bitumen-/solid-wetting phenomena is still very limited. The current study aims to reveal how the mineralogy and topography of aggregate surfaces affect the wetting and water-induced dewetting of bitumen on aggregate surfaces. For this, contact angle tests were performed to measure the surface energies of bitumen and aggregate surfaces varying in both mineralogy and roughness. Based on the measurements, both qualitative and quantitative analyses were conducted for the interaction of bitumen/aggregate interface in air and water environments. Complete wetting and complete dewetting hold for the air/bitumen/aggregate and water/bitumen/aggregate interfaces, respectively. The negative interfacial adhesive energy for the air/bitumen/aggregate interface and the interfacial debonding energy for the water/bitumen/aggregate interface imply that both bitumen wetting and water-induced bitumen dewetting on flat surfaces are thermodynamically favorable. The Wenzel model approximation holds up for the rough aggregate surface interface systems. The interfacial adhesive energy and interfacial debonding energy are enhanced geometrically by the roughness factor r, which indicates that the textured aggregate surface is in favor of force-induced interfacial cracking resistance but shows an adverse effect to moisture damage resistance. The findings from the current study provide guidelines for materials design in pavement engineering.