Interacting with virtual objects with haptic feedback directly using the user's hand (hand-based haptic interaction) has attracted increasing attention. Due to the high degrees of freedom of the hand, compared with tool-based interactive simulation using a pen-like haptic proxy, hand-based haptic simulation faces greater challenges, mainly including higher motion mapping and modeling difficulty of deformable hand avatars, higher computational complexity of contact dynamics, and nontrivial multi-modal fusion feedback. In this article, we aim to review key computing components for hand-based haptic simulation, and draw out major findings in this direction while analyzing the gaps toward immersive and natural hand-based haptic interaction. To this end, we investigate existing relevant studies on hand-based interaction with kinesthetic and/or cutaneous display in terms of virtual hand modeling, hand-based haptic rendering, and visuo-haptic fusion feedback. By identifying current challenges, we finally highlight future perspectives in this field.