Spectroscopy in the terahertz frequency regime is a sensitive tool to probe solvation-induced effects in aqueous solutions. Yet, a systematic understanding of spectral lineshapes as a result of distinct solvation contributions remains terra incognita. We demonstrate that modularization of amino acids in terms of functional groups allows us to compute their distinct contributions to the total terahertz response. Introducing the molecular cross-correlation analysis method provides unique access to these site-specific contributions. Equivalent groups in different amino acids lead to look-alike spectral contributions, whereas side chains cause characteristic but additive complexities. Specifically, hydrophilic solvation of the zwitterionic groups in valine and glycine leads to similar terahertz responses which are fully decoupled from the side chain. The terahertz response due to H-bonding within the large hydrophobic solvation shell of valine turns out to be nearly indistinguishable from that in bulk water in direct comparison to the changes imposed by the charged functional groups that form strong H-bonds with their hydration shells. Thus, the hydrophilic groups and their solvation shells dominate the terahertz absorption difference, while on the same intensity scale, the influence of hydrophobic water can be neglected.