Structural characterization of the interaction between histone tails and effector modules (bromodomains, chromodomains, PHD fingers, etc.) is fundamental to understand the mechanistic aspects of epigenetic regulation of gene expression. In recent years many researchers have applied this approach to specific systems, thus providing a valuable but fragmentary view of the histone-effector interaction. In our work we use this information to characterize the structural features of the two main components of this interaction, histone peptides and the binding site of effector domains (focusing on those which target modified lysines), and increase our knowledge on its specificity determinants. Our results show that the binding sites of effectors are structurally variable, but some clear trends allow their classification in three main groups: flat-groove, narrow-groove and cavity-insertion. In addition, we found that even within these classes binding site variability is substantial. These results in context with the work from other researchers indicate that the there are at least two determinants of binding specificity in the binding site of effector modules. Finally, our analysis of the histone peptides sheds light on the structural transition experienced by histone tails upon effector binding, showing that it may vary depending on the local properties of the sequence stretch considered, thus allowing us to identify an additional specificity determinant for this interaction. Overall, the results of our analysis contribute to clarify the origins of specificity: different regions of the binding site and, in particular, differences in the disorder-order transitions experienced by different histone sequence stretches upon binding.