For atomically thin two-dimensional materials, variations in layer thickness can result in significant changes in the electronic energy band structure and physicochemical properties, thereby influencing the carrier dynamics and device performance. In this work, we employ time- and energy-resolved photoemission electron microscopy to reveal the ultrafast carrier dynamics of PdSe2 with different layer thicknesses. We find that for few-layer PdSe2 with a semiconductor phase, an ultrafast hot carrier cooling on a timescale of approximately 0.3 ps and an ultrafast defect trapping on a timescale of approximately 1.3 ps are unveiled, followed by a slower decay of approximately tens of picoseconds. However, for bulk PdSe2 with a semimetal phase, only an ultrafast hot carrier cooling and a slower decay of approximately tens of picoseconds are observed, while the contribution of defect trapping is suppressed with the increase of layer number. Theoretical calculations of the electronic energy band structure further confirm the transition from a semiconductor to a semimetal. Our work demonstrates that TR- and ER-PEEM with ultrahigh spatiotemporal resolution and wide-field imaging capability has great advantages in revealing the intricate details of ultrafast carrier dynamics of nanomaterials.