Influenza A viruses (IAV) are widespread in birds and domestic poultry, occasionally causing severe epidemics in humans and posing health threats. Hence, the need to develop a strategy for prophylaxis or therapy, such as a broadly neutralizing antibody against IAV, is urgent. In this study, single-chain variable fragment (scFv) phage display technology was used to select scFv fragments recognizing influenza envelope proteins. The Tomlinson I and J scFv phage display libraries were screened against the recombinant HA2 protein (rHA2) for three rounds. Only the third-round elution sample of the Tomlinson J library showed high binding affinity to rHA2, from which three clones (3JA18, 3JA62, and 3JA78) were chosen for preparative-scale production as soluble antibody by E. coli. The clone 3JA18 was selected for further tests due to its broad affinity for influenza H1N1, H3N2 and H5N1. Simulations of the scFv 3JA18-HA trimer complex revealed that the complementarity-determining region of the variable heavy chain (VH-CDR2) bound the stem region of HA. Neutralization assays using a peptide derived from VH-CDR2 also supported the simulation model. Both the selected antibody and its derived peptide were shown to suppress infection with H5N1 and H1N1 viruses, but not H3N2 viruses. The results also suggested that the scFvs selected from rHA2 could have neutralizing activity by interfering with the function of the HA stem region during virus entry into target cells.