The time-domain passivity approach (TDPA) has gained considerable attention (even beyond the fields of haptics and teleoperation) because it can ensure passivity while maintaining high system performance. Despite its simplicity and effectiveness, the practical applications of this method are limited by its high-frequency force modification, so-called chattering. Virtual mass-spring (VMS) filters have been proposed to overcome this limitation. However, such approaches also suffer from certain limitations attributable to the distorted forces. Herein, we propose a method to mitigate the chattering of the TDPA, while also retaining its merits. We introduce a non-zero velocity threshold, within which the adaptive damping of the TDPA is scaled down, to attenuate the chattering while maintaining a stable response significantly. The feasibility and effectiveness of the proposed method are evaluated using a time-delayed bilateral teleoperation system. We confirm that the proposed method can significantly outperform the conventional and VMS filter-based TDPAs.