A high-sensitivity vector magnetic field is proposed and demonstrated. The sensor consists of a side-polished-fiber (SPF)-based surface plasmon resonance (SPR) structure integrated with ferrofluid. Because of the high refractive index sensitivity of the SPR scheme and the outstanding magneto optical properties of ferrofluid, the sensor shows a high sensitivity (up to 598.7 pm/Oe) to magnetic field intensity. Moreover, owing to the non-circular-symmetric geometry of the SPF and non-uniform distribution of the ferrofluid around the SPF, the sensor exhibits a sensitivity of -5.63 nm/deg to the orientation of the magnetic field. The proposed vector magnetic field sensor, integrating over magnetic, plasmonic, and fiber-optic schemes, is highly sensitive to both the intensity and orientation of the magnetic field simultaneously, and holds potential in applications in many fields, such as medicine, industry, and the military.