We report an all-fiber passively mode-locked laser based on a saturable absorber fabricated by filling short single-wall carbon nanotubes into cladding holes of grapefruit-type photonic crystal fiber. The single-wall carbon nanotube is insensitive to polarization of light for its one-dimensional structure, which suppresses the polarization dependence loss. Carbon nanotubes interact with photonic crystal fiber with ultra-weak evanescent field, which enhances the damage threshold of the saturable absorber and improves the operating stability. In our experiment, conventional soliton with a pulse duration of 1.003 ps and center wavelength of 1566.36 nm under a pump power of 240 mW is generated in a compact erbium-doped fiber laser cavity with net anomalous dispersion of -0.4102 ps2. The signal to noise ratio of the fundamental frequency component is ~80 dB. The maximum average output power of the mode-locked laser reaches 9.56 mW under a pump power of 360 mW. The output power can be further improved by a higher pump power.