All-polymer solar cells (all-PSCs), based on p-type polymer donors and n-type acceptors as the active layer, offer exceptional promise because of excellent thermal stability, superior film formation, and good mechanical stress as a unique bulk heterojunction (BHJ) solar cell combination. Therefore, tuning the molecular composition between polymers is crucial for optimizing power conversion efficiency (PCE) in these all-PSC systems. In this study, we synthesized a series of naphthalene diimide (NDI)-based random terpolymers P(NDI-BDD10), P(NDI-TPD10), P(NDI-TT10), and P(NDI-2FQ10) with axisymmetric (BDD, TPD) and asymmetric (TT, 2FQ) electron acceptors. Compared with the blend morphology of PBDB-T:N2200, their diverse effects due to the addition of trace amounts of axisymmetric and asymmetric components were comprehensively investigated using physical and surface analyses and structural simulations. Consequently, most of our polymer acceptors demonstrated improved fill factors (FFs) in the optimal morphology. P(NDI-BDD10)-based devices achieved the highest PCE of 6.80% and FF of 69.1%, while the architecturally most asymmetric P(NDI-TT10)-based devices reached the lowest PCE of 4.52% despite an enhanced FF of 65.4%. As a result, the appropriate molecular arrangement is crucial for obtaining the desired morphology and improved PCE. Our findings give novel molecular design insight into the distinctions between axisymmetric and asymmetric electron acceptors and seem significant for achieving improved morphological features and efficiency.