We propose a constitutive model to describe the nonlocality, hysteresis, and several flow features of dry granular materials. Taking the well-known inertial number I as a measure of sheared-induced local fluidization, we derive a relaxation model for I according to the evolution of microstructure during avalanche and dissipation processes. The model yields a nonmonotonic flow law for a homogeneous flow, accounting for hysteretic solid-fluid transition and intermittency in quasistatic flows. For an inhomogeneous flow, the model predicts a generalized Bagnold shear stress revealing the interplay of two microscopic nonlocal mechanisms: collisions among correlated structures and the diffusion of fluidization within the structures. In describing a uniform flow down an incline, the model reproduces the hysteretic starting and stopping heights and the Pouliquen flow rule for mean velocity. Moreover, a dimensionless parameter reflecting the nonlocal effect on the flow is discovered, which controls the transition between Bagnold and creeping flow dynamics.