Recent advances in automation technology can lead to unsafe situations where operators lose their sense of agency over the automated equipment. On the other hand, increasing evidence has shown that providing operators with opportunities of continuous operation and helping them improve their performance on tasks through automation can boost their sense of agency. However, it is challenging to ensure that the operator maintains a sense of agency when working with a fully automated tool that removes him/her from the control loop. By demonstrating a tracking task in which participants continuously tracked a moving target through a cursor controlled by a joystick under different levels of automation, we illustrate how the participants' sense of agency and tracking performance were altered in accordance with the level of automation. The results showed that their sense of agency was enhanced by increasing automation but began to decline when the level of automation exceeded 90%. More generally, this suggests that allowing operators a little contribution to control over the continuous operation of an automated tool may be sufficient to maintain their sense of agency while yielding the maximum improvement in performance.