Understanding the specificities of human serum antibodies that broadly neutralize HIV can inform prevention and treatment strategies. Here we describe a deep mutational scanning system that can measure the effects of combinations of mutations to HIV envelope (Env) on neutralization by antibodies and polyclonal serum. We first show that this system can accurately map how all functionally tolerated mutations to Env affect neutralization by monoclonal antibodies. We then comprehensively map Env mutations that affect neutralization by a set of human polyclonal sera known to target the CD4-binding site that neutralize diverse strains of HIV. The neutralizing activities of these sera target different epitopes, with most sera having specificities reminiscent of individual characterized monoclonal antibodies, but one sera targeting two epitopes within the CD4 binding site. Mapping the specificity of the neutralizing activity in polyclonal human serum will aid in assessing anti-HIV immune responses to inform prevention strategies.