State-of-the-art technologies for hand (and finger) motion tracking do not always provide accurate and robust tracking. For example, severe occlusions can affect tracking with vision sensors, electromagnetic interference affects tracking with inertial measurement units (IMUs) and compasses, and ambiguous mechanical contact can affect tracking with soft sensors (i.e., the inability to distinguish motion-induced deformation). Here, we report a visual-inertial skeleton tracking (VIST) framework that provides robust and accurate hand tracking in a variety of real-world scenarios. Our proposed VIST framework comprises a sensor glove with multiple IMUs and passive visual markers as well as a head-mounted stereo camera. VIST also uses a tightly coupled filtering-based visual-inertial fusion algorithm to estimate the hand/finger motion and autocalibrates hand/glove-related kinematic parameters simultaneously while taking into account the hand anatomical constraints. Our VIST framework exhibits good tracking accuracy and robustness, affordable material cost, lightweight hardware and software, and durability to permit washing. We validate our VIST framework through quantitative and qualitative experiments in real-world conditions. Our approach to hand tracking has the potential to enrich not only human-robot interaction applications (e.g., direct humanoid hand teleoperation, hand-based collaborative robot programming, and drone swarm control) but also the user experience in many virtual reality and augmented reality applications.