Internet of Health Things (IoHT) is a promising e-Health paradigm that involves offloading numerous computational-intensive and delay-sensitive tasks from locally limited IoHT points to edge servers (ESs) with abundant computational resources in close proximity. However, existing computation offloading techniques struggle to meet the burgeoning health demands in ultra-reliable and low-latency communication (URLLC), one of the 5G application scenarios. This paper proposes a Multi-Agent Soft-Actor-Critic-discrete based URLLC-constrained task offloading and resource allocation (MASACDUA) scheme to maximize throughput while minimizing power consumption on the remote side, considering the long-term URLLC constraints. The URLLC constraint conditions are formulated using extreme value theory, and Lyapunov optimization is employed to divide the problem into task offloading and computation resource allocation. MASAC-discrete and a queue backlog-aware algorithm are utilized to approach task offloading and computation resource allocation, respectively. Extensive simulation results demonstrate that MASACDUA outperforms traditional DRL algorithms under different IoHT points and data arrival rate intervals and achieves superior performance in delay, bound violation probability, and other characteristics related to URLLC.