The apolipoprotein E (ApoE) signal peptide is a short stretch of N-terminal amino acids that direct the ApoE protein to the endoplasmic reticulum after synthesis. Previous studies have shown that this peptide can bind to lipid membranes in a cholesterol-dependent manner; however, the mechanism of this interaction is yet to be clarified. In this study, we aimed to investigate how the composition of neighboring lipids affects the membrane-binding of the ApoE signal peptide. We found that a negatively charged lipid, such as phosphatidylglycerol, can act as a switch that reduces the binding efficiency of the peptide to cholesterol-rich membranes. Interestingly, phosphatidylethanolamine does not activate the cholesterol-dependent binding of the ApoE signal peptide yet acts synergistically to enhance the cholesterol sensitivity in phosphatidylglycerol-containing membranes. To the best of our knowledge, this is the first report of modulation of the affinity of a peptide for a membrane by a neighboring lipid rather than by the lipid-binding domain of the peptide. Our findings revealed a novel role of lipid diversity in modulating the membrane binding of the ApoE signal peptide and its potential implications in the unidirectional trafficking of a newly synthesized protein from the ribosomes to the endoplasmic reticulum.