Active chiral viscoelastic materials exhibit elastic responses perpendicular to the applied stresses, referred to as odd elasticity. We use a covariant formulation of viscoelasticity combined with an entropy production analysis to show that odd elasticity is not only present in active systems but also in broad classes of passive chiral viscoelastic fluids. In addition, we demonstrate that linear viscoelastic chiral solids require activity in order to manifest odd elastic responses. To model the phenomenon of passive odd viscoelasticity we propose a chiral extension of Jeffreys model. We apply our covariant formalism in order to derive the dispersion relations of hydrodynamic modes and obtain clear imprints of odd viscoelastic behavior.