In this paper, we proposed and demonstrated two kinds of all few-mode fiber lasers with self-starting high-order mode (HOM) oscillation. The fundamental mode can be completely suppressed by using a bandpass filter with a few-mode fiber pigtail. In the continuous-wave (CW) regime, the fiber laser directly oscillates in HOM with a signal-to-noise ratio as high as 70 dB, and the slope efficiency is up to 46%. The self-starting HOM mode-locked pulse can be easily achieved by employing a saturable absorber. The HOM oscillation pulsed fiber laser stably operates at 1063.72 nm with 3dB of 0.05 nm, which can deliver cylindrical vector beams with a high mode purity of over 98%. To our knowledge, this is the first demonstration for self-starting HOM direct oscillation in stable CW and pulsed operation states without additional adjustment. This compact and stable HOM fiber laser with a simple structure can have important applications in materials processing, optical trapping, and spatiotemporal nonlinear optics. Moreover, this work may offer a promising approach to realizing high-power fiber laser with arbitrary HOMs stable output.