A lateral deviation of the walking trajectory is often observed in stroke patients with unilateral spatial neglect. However, existing research appears to be contradictory regarding the direction of this deviation. The aim of the present study was to gain more insight into the walking trajectory of neglect patients. Twelve right hemisphere stroke patients (six neglect, six no neglect), eight left hemisphere stroke patients (none neglect) and 10 healthy control subjects were instructed to walk towards a target while a two-dimensional ultrasonic positioning system recorded their walking trajectory. Patients' recovery of walking ability was assessed and they were tested for the presence of neglect. Neglect patients showed a larger lateral deviation in their walking trajectory compared to stroke patients without neglect or controls. Neglect patients with good walking ability showed a deviation to the contralesional side. Neglect patients with limited walking ability showed a deviation to the ipsilesional side. Within the neglect group we found no relation between the severity of neglect and lateral deviation. Differences in walking ability may account for the contradictory results between studies regarding the lateral deviation in neglect patients' walking trajectory. We suggest that when a neglect patient's walking ability is limited, walking towards a target becomes a dual task: heading control and walking. A limited walking ability will cause a higher task priority of walking compared to heading control. This shift in task priority may be causing the change in walking trajectory deviation.