Optical motion-capture systems can be used to animate characters in real time based on human demonstrations. However, most approaches do not consider the detailed finger movements necessary for object manipulation. The proposed online motion-capture framework encourages natural motions by automatically guiding the avatar's motion toward the desired behavior based on a set of captured reference motions for the same behavior. The proposed motion controller imitates the user's finger and body gestures to enable performance-based animation of human avatars that can manipulate virtual objects in real time.