The Acoustic Complexity Index (ACI) is increasingly applied to the study of biodiversity in aquatic habitats. However, it remains unknown which types of acoustic information are highlighted by this index in underwater environments. This study explored the robustness of the ACI to fine variations in fish sound abundance (i.e. number of sounds) and sound diversity (i.e. number of sound types) in field recordings and controlled experiments. The ACI was found to be sensitive to variations in both sound abundance and sound diversity, making it difficult to discern between these variables. Furthermore, the ACI was strongly dependent on the settings used for its calculation (i.e. frequency and temporal resolution of the ACI algorithm, amplitude filter). Care should thus be taken when comparing ACI absolute values between studies, or between sites with site-specific characteristics (e.g. species diversity, fish vocal community composition). As the use of ecoacoustic indices presents a promising tool for the monitoring of vulnerable environments, methodological validations like those presented in this paper are of paramount importance in understanding which biologically important information can be gathered by applying acoustic indices to Passive Acoustic Monitoring data.