Numerous benchmark datasets and evaluation toolkits have been designed to facilitate visual object tracking evaluation. However, it is not clear which evaluation protocols are preferred for different tracking objectives. Even worse, different evaluation protocols sometimes yield contradictory conclusions, further hampering reliable evaluation. Therefore, we 1) introduce the new concept of mirror tracking to measure the robustness of a tracker and identify its over-fitting scenarios; 2) measure the robustness of the evaluation ranks produced by different evaluation protocols; and 3) report a detailed analysis of milestone tracking challenges, indicating their application scenarios. Our experiments are based on two state-of-the-art challenges, namely, OTB and VOT, using the same trackers and datasets. Based on the experiments, we conclude that 1) the proposed mirror tracking metrics can identify the over-fitting scenarios of a tracker, 2) the ranks produced by OTB are more robust than those produced by VOT, and 3) the joint ranks produced by OTB and VOT can be used to measure failure recovery.