Protein-protein interactions that include recognition of short sequences of amino acids, or peptides, are critical in cellular processes. Protein-peptide interaction surface areas are relatively small and shallow, and there are often overlapping specificities in families of peptide-binding domains. Therefore, dissecting selectivity determinants can be challenging. PDZ domains are an example of a peptide-binding domain located in several intracellular signaling and trafficking pathways, which form interactions critical for the regulation of receptor endocytic trafficking, tight junction formation, organization of supramolecular complexes in neurons, and other biological systems. These domains are also directly targeted by pathogens, and a hallmark of many oncogenic viral proteins is a PDZ-binding motif. However, amidst sequences that target PDZ domains, there is a wide spectrum in relative promiscuity. For example, the viral HPV16 E6 oncoprotein recognizes over double the number of PDZ domain-containing proteins as the cystic fibrosis transmembrane conductance regulator (CFTR) in the cell, despite similar PDZ targeting-sequences and identical motif residues. Here, we determine binding affinities for PDZ domains known to bind either HPV16 E6 alone or both CFTR and HPV16 E6, using peptides matching WT and hybrid sequences. We also use energy minimization to model PDZ-peptide complexes and use sequence analyses to investigate this difference. We find that while the majority of single mutations had a marginal effect on overall affinity, the additive effect on the free energy of binding accurately describes the selectivity observed. Taken together, our results describe how complex and differing PDZ interactomes can be programmed in the cell.