The environmental assessment of potential effects of contaminated harbor sediments stabilized with hydraulic binders and the determination of remediation endpoints require the determination of pollutants leaching potentials. Moreover, little information about the speciation and mobility of inorganic contaminants in these specific solid matrices is available in the literature. The objective of this paper is to investigate the relationship between mineralogy and leachability of contaminants (copper, lead, and zinc) present in a French harbor sediment stabilized with quicklime and Portland cement. Batch equilibrium leaching tests at various pH, chemical analysis of leachates, and mineralogical studies (X-ray diffraction, scanning electron microscopy-energy dispersive spectroscopy, and diffuse reflectance infrared Fourier transform) have been combined in the present investigation. The acid neutralization capacity of the stabilized matrix studied is first controlled by the dissolution of portlandite (pH ~12), followed by the dissolution of C-S-H (pH ~11) and the dissolution of ettringite (pH ~10). Finally, a very high buffering capacity of this stabilized sediment is observed for pH values around 6. This equilibrium is mainly controlled by the dissolution of iron sulfides and carbonate minerals. Consequently, the mobilization of inorganic contaminants as a function of pH remains very low (<0.1 wt%) for pH values above 6 and significantly increases for pH below these values. This research confirms the importance of a combined methodology for the intrinsic characterization of potential mobilization of contaminants in a stabilized sediment and for a better understanding of geochemical processes that affect contaminant fate, transformation, and transport in the subsurface environment.