The hemagglutinin (HA) protein of the influenza virus binds to the host cell receptor in the early stage of viral infection. A change in binding specificity from avian 2-3 to human 2-6 receptor is essential for optimal human-to-human transmission and pandemics. Therefore, it is important to reveal the key factors governing the binding affinity of HA-receptor complex at the molecular level for the understanding and prediction of influenza pandemics. In this work, on the basis of ab initio fragment molecular orbital (FMO) method, we have carried out the interaction energy analysis of HA-receptor complexes to quantitatively elucidate the binding specificity of HAs to avian and human receptors. To discuss the binding property of influenza HA comprehensively, a number of HAs from human H1, swine H1, avian H3 and avian H5 viruses were analyzed. We performed detailed investigations about the interaction patterns of complexes of various HAs and receptor analogues, and revealed that intra-molecular interactions between conserved residues in HA play an important role for HA-receptor binding. These results may provide a hint to understand the role of conserved acidic residues at the receptor binding site which are destabilized by the electrostatic repulsion with sialic acid. The calculated binding energies and interaction patterns between receptor and HAs are consistent with the binding specificities of each HA and thus explain the receptor binding mechanism. The calculated results in the present analysis have provided a number of viewpoints regarding the models for the HA-receptor binding specificity associated with mutated residues. Examples include the role of Glu190 and Gln226 for the binding specificity of H5 HA. Since H5 HA has not yet been adapted to human receptor and the mechanism of the specificity change is unknown, this result is helpful for the prediction of the change in receptor specificity associated with forthcoming possible pandemics.