Recently, the switching of a perpendicularly magnetized ferromagnet (FM) by injecting an in-plane current into an attached non-magnet (NM) has become of emerging technological interest. This magnetization switching is attributed to the spin-orbit torque (SOT) originating from the strong spin-orbit coupling of the NM layer. However, the switching efficiency of the NM/FM structure itself may be insufficient for practical use, as for example, in spin transfer torque (STT)-based magnetic random access memory (MRAM) devices. Here we investigate spin torque in an NM/FM structure with an additional spin polarizer (SP) layer abutted to the NM layer. In addition to the SOT contribution, a spin-polarized current from the SP layer creates an extra spin chemical potential difference at the NM/FM interface and gives rise to a STT on the FM layer. We show that, using typical parameters including device width, thickness, spin diffusion length, and the spin Hall angle, the spin torque from the SP layer can be much larger than that from the spin Hall effect (SHE) of the NM.