Translation initiation is a major rate-limiting step for protein synthesis. However, recent studies strongly suggest that the efficiency of protein synthesis is additionally regulated by multiple factors that impact the elongation phase. To assess the influence of early elongation on protein synthesis, we employed a library of more than 250,000 reporters combined with in vitro and in vivo protein expression assays. Here we report that the identity of the amino acids encoded by codons 3 to 5 impact protein yield. This effect is independent of tRNA abundance, translation initiation efficiency, or overall mRNA structure. Single-molecule measurements of translation kinetics revealed pausing of the ribosome and aborted protein synthesis on codons 4 and 5 of distinct amino acid and nucleotide compositions. Finally, introduction of preferred sequence motifs only at specific codon positions improves protein synthesis efficiency for recombinant proteins. Collectively, our data underscore the critical role of early elongation events in translational control of gene expression.