Spin orbit torque (SOT) has drawn widespread attention in the emerging field of magnetic memory devices, such as magnetic random access memory (MRAM). To promote the performance of SOT-MRAM, most efforts have been devoted to enhance the SOT switching efficiency by improving the damping-like torque. Recently, some studies noted that the field-like torque also plays a crucial role in the nanosecond-timescale SOT dynamics. However, there is not yet an effective way to tune its relative amplitude. Here, we experimentally modulate the field-like SOT in W/CoFeB/MgO trilayers through tuning the interfacial spin accumulation. By performing spin Hall magnetoresistance measurement, we find that the CoFeB with enhanced spin dephasing, either generated from larger layer thickness or from proper annealing, can distinctly boost the spin absorption and enhance the interfacial spin mixing conductance Gr. While the damping-like torque efficiency increases with Gr, the field-like torque efficiency is found to decrease with it. The results suggest that the interfacial spin accumulation, which largely contributes to the field-like torque, is reduced by higher interfacial spin transparency. Our work shows a new path to further improve the performance of SOT-based ultrafast magnetic devices.