A near-infrared spectrometer based on offset fused multimode fiber (MMF) is investigated in this study. The light spectrum is recovered by analyzing the speckle images when light is passing through the MMF. In order to generate adequate speckles, a polarization maintaining fiber (PMF) and a 30 cm long MMF are fused with a vertical offset. Seven different offset displacements are implemented in the fiber fusion. The follow-up experiments show that the fiber offset fusion has a significant influence on the spectral correlation and the resolution. Larger offset fusion can excite more high-order modes in the MMF, and it greatly improves the spectrometer's performance. The simulation results also show that more modes are excited in MMF, and the increase of mode number leads to lower correlation coefficients of the neighboring spectral channels. However, large offset fusion increases the fusion and the insertion loss of the whole system, which may bring difficulties in the low-light cases. In addition, an image denoising algorithm based on dynamic threshold filtering and a spectral reconstruction algorithm originated from complete orthogonal decomposition were used to remove the speckle pattern noise and recover the spectrum. The final speckle-based spectrometer has a spectral resolution of 0.6∼0.016nm depending on the different offset fusions.