Nonlinear optical effects in layered two-dimensional transition metal chalcogenides have been extensively explored recently because of the promising prospect of the nonlinear optical effects for various optoelectronic applications. However, these materials possess sizable bandgaps ranging from visible to ultraviolet region, so the investigation of narrow-bandgap materials remains deficient. Here, we report our comprehensive study on the nonlinear optical processes in palladium diselenide (PdSe2) that has a near-infrared bandgap. Interestingly, this material exhibits a unique thickness-dependent second harmonic generation feature, which is in contrast to other transition metal chalcogenides. Furthermore, the two-photon absorption coefficients of 1-3 layer PdSe2 (β ~ 4.16 × 105, 2.58 × 105, and 1.51 × 105 cm GW-1) are larger by two and three orders of magnitude than that of the conventional two-dimensional materials, and giant modulation depths (αs ~ 32%, 27%, and 24%) were obtained in 1-3 layer PdSe2. Such unique nonlinear optical characteristics make PdSe2 a potential candidate for technological innovations in nonlinear optoelectronic devices.