Potassium channel dysfunction underlies diseases such as epilepsy, hypertension, cardiac arrhythmias, and multiple sclerosis. Neurotoxins that selectively inhibit potassium channels, alpha-KTx, have provided invaluable information for dissecting the contribution of different potassium channels to neurotransmission, vasoconstriction, and lymphocyte proliferation. Thus, alpha-KTx specificity comprises an important first step in potassium channel-directed drug discovery for these diseases. Despite extensive functional and structural studies of alpha-KTx-potassium channel complexes, none have predicted the molecular basis of alpha-KTx specificity. Here we show that by minimizing the differences in binding free energy between selective and nonselective alpha-KTx we are able to identify all of the determinants of alpha-KTx specificity for calcium-activated versus voltage-dependent potassium channels. Because these determinants correspond to unique features of the two types of channels, they provide a way to develop more accurate models of alpha-KTx-potassium channel complexes that can be used to design novel selective alpha-KTx inhibitors.