We consider the problem of building a continuous stochastic model, i.e., a Langevin or Fokker-Planck equation, through a well-controlled coarse-graining procedure. Such a method usually involves the elimination of the fast degrees of freedom of the "bath" to which the particle is coupled. Specifically, we look into the general case where the bath may be at negative temperatures, as found, for instance, in models and experiments with bounded effective kinetic energy. Here, we generalize previous studies by considering the case in which the coarse graining leads to (i) a renormalization of the potential felt by the particle, and (ii) spatially dependent viscosity and diffusivity. In addition, a particular relevant example is provided, where the bath is a spin system and a sort of phase transition takes place when going from positive to negative temperatures. A Chapman-Enskog-like expansion allows us to rigorously derive the Fokker-Planck equation from the microscopic dynamics. Our theoretical predictions show excellent agreement with numerical simulations.