Accurate prediction of the protein-ligand binding affinity (PLBA) with an affordable cost is one of the ultimate goals in the field of structure-based drug design (SBDD), as well as a great challenge in the computational and theoretical chemistry. Herein, we have systematically addressed the complicated solvation and desolvation effects on the PLBA brought by the difference of the explicit water in the protein cavity before and after ligands bind to the protein-binding site. Based on the new solvation model, a nonfitting method at the first-principles level for the PLBA prediction was developed by taking the bridging and displaced water (BDW) molecules into account simultaneously. The newly developed method, DOX_BDW, was validated against a total of 765 noncovalent and covalent protein-ligand binding pairs, including the CASF2016 core set, Cov_2022 covalent binding testing set, and six testing sets for the hit and lead compound optimization (HLO) simulation. In all of the testing sets, the DOX_BDW method was able to produce PLBA predictions that were strongly correlated with the corresponding experimental data (R = 0.66-0.85). The overall performance of DOX_BDW is better than the current empirical scoring functions that are heavily parameterized. DOX_BDW is particularly outstanding for the covalent binding situation, implying the need for considering an electronic structure in covalent drug design. Furthermore, the method is especially recommended to be used in the HLO scenario of SBDD, where hundreds of similar derivatives need to be screened and refined. The computational cost of DOX_BDW is affordable, and its accuracy is remarkable.