The spike glycoprotein (S-protein) mediates SARS-CoV-2 entry via intermolecular interaction with human angiotensin-converting enzyme 2. The receptor binding domain (RBD) of the S-protein has been considered critical for this interaction and acts as the target of numerous neutralizing antibodies and antiviral peptides. This study used the fragment molecular orbital method to analyze the interactions between the RBD and antibodies/peptides and extracted crucial residues that can be used as epitopes. The interactions evaluated as interfragment interaction energy values between the RBD and 12 antibodies/peptides showed a fairly good correlation with the experimental activity pIC50 (R2 = 0.540). Nine residues (T415, K417, Y421, F456, A475, F486, N487, N501, and Y505) were confirmed as being crucial. Pair interaction energy decomposition analyses showed that hydrogen bonds, electrostatic interactions, and π-orbital interactions are important. Our results provide essential information for understanding SARS-CoV-2-antibody/peptide binding and may play roles in future antibody/antiviral drug design.