The ligand binding pockets of proteins have preponderance of hydrophobic amino acids and are typically within the apolar interior of the protein; nevertheless, they are able to bind low complexity, polar, water-soluble fragments. In order to understand this phenomenon, we analyzed high resolution X-ray data of protein-ligand complexes from the Protein Data Bank and found that fragments bind to proteins with two near optimal geometry H-bonds on average. The linear extent of the fragment binding site was found not to be larger than 10 Å, and the H-bonding region was found to be restricted to about 5 Å on average. The number of conserved H-bonds in proteins cocrystallized with multiple different fragments is also near to 2. These fragment binding sites that are able to form limited number of strong H-bonds in a hydrophobic environment are identified as hot spots. An estimate of the free-energy gain of H-bond formation versus apolar desolvation supports that fragment sized compounds need H-bonds to achieve detectable binding. This suggests that fragment binding is mostly enthalpic that is in line with their observed binding thermodynamics documented in Isothermal Titration Calorimetry (ITC) data sets and gives a thermodynamic rationale for fragment based approaches. The binding of larger compounds tends to more rely on apolar desolvation with a corresponding increase of the entropy content of their binding free-energy. These findings explain the reported size-dependence of maximal available affinity and ligand efficiency both behaving differently in the small molecule region featured by strong H-bond formation and in the larger molecule region featured by apolar desolvation.