Since entering the human population, Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2; the causative agent of Coronavirus Disease 2019 [COVID-19]) has spread worldwide, causing >100 million infections and >2 million deaths. While large-scale sequencing efforts have identified numerous genetic variants in SARS-CoV-2 during its circulation, it remains largely unclear whether many of these changes impact adaptation, replication, or transmission of the virus. Here, we characterized 14 different low-passage replication-competent human SARS-CoV-2 isolates representing all major European clades observed during the first pandemic wave in early 2020. By integrating viral sequencing data from patient material, virus stocks, and passaging experiments, together with kinetic virus replication data from nonhuman Vero-CCL81 cells and primary differentiated human bronchial epithelial cells (BEpCs), we observed several SARS-CoV-2 features that associate with distinct phenotypes. Notably, naturally occurring variants in Orf3a (Q57H) and nsp2 (T85I) were associated with poor replication in Vero-CCL81 cells but not in BEpCs, while SARS-CoV-2 isolates expressing the Spike D614G variant generally exhibited enhanced replication abilities in BEpCs. Strikingly, low-passage Vero-derived stock preparation of 3 SARS-CoV-2 isolates selected for substitutions at positions 5/6 of E and were highly attenuated in BEpCs, revealing a key cell-specific function to this region. Rare isolate-specific deletions were also observed in the Spike furin cleavage site during Vero-CCL81 passage, but these were rapidly selected against in BEpCs, underscoring the importance of this site for SARS-CoV-2 replication in primary human cells. Overall, our study uncovers sequence features in SARS-CoV-2 variants that determine cell-specific replication and highlights the need to monitor SARS-CoV-2 stocks carefully when phenotyping newly emerging variants or potential variants of concern.