Peptide-mediated interactions are crucial to a variety of functions in the living cell and are estimated to be involved in up to 40 % of all cellular processes. Fast and reliable inference of such interactions is fundamentally important for our understanding and, then, reconstruction of complete virtual interactomics involved in a specific cell, tissue or organism. In the current study, we performed structure-level characterization, modeling and prediction of protein-peptide recognition specificity and stability in a high-throughput manner. To achieve this, the classical chemometrics methodology quantitative structure-activity relationship (QSAR), which is traditionally applied to small-molecule entities such as drug compounds and environmental chemicals, was employed to statistically correlate structure features with binding affinities for a panel of structure-solved, affinity-known protein-peptide complexes compiled from the PDB database and literatures. In the standard QSAR procedure, various structural descriptors including physicochemical, geometrical and constitutional parameters that characterize diverse aspects of protein-peptide interaction property were derived from the biomacromolecular complex structure architecture, and these descriptors were then correlated with experimentally measured affinities by using the partial least squares (PLS) regression and Gaussian process (GP) in conjunction with genetic algorithm (GA) variable selection. The nonlinear GA/GP method was found to perform much well as compared to linear GA/PLS modeling, suggesting that the protein-peptide interaction system is highly complicated that may involve strong noise and interactive effect. The optimal GA/GP model revealed that the interface size and solvent effect play a critical role in protein-peptide binding, and other properties such as peptide length and flexibility also contribute significantly to the binding. A further test on 2,018 human amphiphysin SH3 domain-binding peptides demonstrated that the purposed QSAR modeling procedure is very fast and effective, which can thus be readily used to perform proteome-wide inference of peptide-mediated interactions.