In this work, we study the dynamic robustness of an endoreversible Carnot cycle working at the maximum per-unit-time performance regime, based on the linearization technique for dynamical systems and the local stability analysis. Our analysis is focused on the endoreversible Carnot refrigerator model, which works in the maximum per-unit-time coefficient of performance. At the steady-state of the maximum performance, the expressions of the relaxation times describing the stability of the system are derived. It is found that the relaxation times in the cycle condition are the function of thermal conductances σh and σc, the temperatures of the heat reservoirs Th and Tc, and the heat capacity C. The influence of the temperature ratio τ = Tc/Th and the thermal conductance ratio σr = σh/σc on the relaxation times is discussed in detail. The results obtained here are useful and provide a potential guidance for the design of an endoreversible Carnot refrigerator working in the maximum performance per cycle time optimization condition.