A custom designed markerless tracking system was demonstrated to be applicable for positron emission tomography (PET) brain imaging. Precise head motion registration is crucial for accurate motion correction (MC) in PET imaging. State-of-the-art tracking systems applied with PET brain imaging rely on markers attached to the patient's head. The marker attachment is the main weakness of these systems. A healthy volunteer participating in a cigarette smoking study to image dopamine release was scanned twice for 2 h with (11)C-racolopride on the high resolution research tomograph (HRRT) PET scanner. Head motion was independently measured, with a commercial marker-based device and the proposed vision-based system. A list-mode event-by-event reconstruction algorithm using the detected motion was applied. A phantom study with hand-controlled continuous random motion was obtained. Motion was time-varying with long drift motions of up to 18 mm and regular step-wise motion of 1-6 mm. The evaluated measures were significantly better for motion-corrected images compared to no MC. The demonstrated system agreed with a commercial integrated system. Motion-corrected images were improved in contrast recovery of small structures.