The relatively recent discovery and characterization of human broadly neutralizing antibodies (bnAbs) against influenza virus provide valuable insights into antiviral and vaccine development. However, the factors that influence the evolution of high-affinity bnAbs remain elusive. We therefore explore the functional sequence space of bnAb C05, which targets the receptor-binding site (RBS) of influenza haemagglutinin (HA) via a long CDR H3. We combine saturation mutagenesis with yeast display to enrich for C05 variants of CDR H3 that bind to H1 and H3 HAs. The C05 variants evolve up to 20-fold higher affinity but increase specificity to each HA subtype used in the selection. Structural analysis reveals that the fine specificity is strongly influenced by a highly conserved substitution that regulates receptor binding in different subtypes. Overall, this study suggests that subtle natural variations in the HA RBS between subtypes and species may differentially influence the evolution of high-affinity bnAbs.