This article generalizes the existing minimal model of the hypothalamic-pituitary-adrenal (HPA) axis in a realistic way, by including memory terms: distributed time delays, on one hand and fractional-order derivatives, on the other hand. The existence of a unique equilibrium point of the mathematical models is proved and a local stability analysis is undertaken for the system with general distributed delays. A thorough bifurcation analysis for the distributed delay model with several types of delay kernels is provided. Numerical simulations are carried out for the distributed delays models and for the fractional-order model with discrete delays, which substantiate the theoretical findings. It is shown that these models are able to capture the vital mechanisms of the HPA system.