This study reports on the direct drive volume control flange-type rotary vane steering gear (DDVC-FRVSG), a promising component with superior advantages of compact structure, powerful vibration absorption, and simple control for application in the controlling course and posture of a vessel. The ability of the DDVC-FRVSG to satisfy the accuracy requirement of the vessel is limited by nonlinear friction. This study proposes two compensation methods to compensate for the nonlinear friction. We establish the mathematical model and the transfer function of the steering gear system and the mathematical model of nonlinear friction on the DDVC-FRVSG system based on the principle of the DDVC-FRVSG. A high-gain proportional-integral-derivative control strategy and another method using the self-adaption robust control strategy is proposed and studied both theoretically and experimentally to suppress the nonlinear friction. With the "no-compensation state" as a benchmark, our measured results by prototype testing has proved that both methods can compensate for the nonlinear friction, with the second method showing a better performance of up to 78.85% increase compared to that of the 41.65% shown by the first one. The outcome of this research will contribute to the rapidity and stability of the DDVC-FRVSG.