Series elastic actuators (SEAs) have widely been adapted in robots where safe human-robot interaction is required for accurate and robust force control. Recent research on the SEAs has shown that the SEA with a user-defined variable stiffness possesses several advantages over the constant stiffness SEA, such as large force range and bandwidth while keeping low output impedance and high force fidelity. However, a limitation of this type of SEA is that an obvious hysteresis effect exists and the associated torque curves are nonlinear and vary with amplitudes. Conventional mathematical hysteresis models are usually developed with some kind of black-box modeling, and the model parameters are adjusted through parameter identification methods. It is challenging to tune the model parameters to match the experimental data well among inputs with different amplitudes, let alone the inverse model of the hysteresis, which is necessary to compensate the hysteresis effect in control. In this paper, a rotary SEA (rSEA) with nonlinear stiffness is proposed. A concept called "virtual deformation" is introduced to mathematically transform the nonlinear curve into a polyline hysteresis model. This eases torque estimation with respect to the deformation of the rSEA. A hysteresis compensation torque controller is implemented for precise torque control. A prototype of the rSEA was fabricated, and the experimental results verified modeling accuracy of the proposed model. Our results showed that, with the new model, the computation cost was greatly reduced while keeping the modeling accuracy almost the same compared with the nonlinear backlash model.