We investigate energy relaxation of hot carriers in monolayer and bilayer graphene devices, demonstrating that the relaxation rate increases significantly as the Dirac point is approached from either the conduction or valence band. This counterintuitive behavior appears consistent with ideas of charge puddling under disorder, suggesting that it becomes very difficult to excite carriers out of these localized regions. These results therefore demonstrate how the peculiar properties of graphene extend also to the behavior of its nonequilibrium carriers.