Exercise scientists frequently use criterion measure tests to assess levels of motor performance and physical fitness or to evaluate the success of their intervention programs. During this process, the same motor test might be administered multiple times. The aim of the present study was to examine whether familiarization with a psychomotor test would influence performance and/or learning. One hundred thirty (n = 130) primary school students participated in the study. Participants were randomly divided into two groups, experimental (n = 60, mean age = 10.74 years old, minimum = 9.08, maximum = 11.83) and control (n = 70, mean age = 10.68 years old, minimum = 9.5, maximum = 11.79). The flamingo balance test was used to examine the postulated hypothesis. Initially, both groups executed the balance test. Then, only the experimental group performed two additional trials, one trial every 3 days. A final trial was administered to all participants, followed by a retention trial 1 week later. The results revealed that both groups improved their performance. The experimental group outperformed the control group. The retention trial showed that performance levels for both groups continued to improve, suggesting that familiarization with the test influenced not only performance but learning as well. Exercise scientists who implement psychomotor tests, such as balance tests, for evaluative purposes should be aware that performance improvement does not necessary represent changes in individuals' performance, but it may reflect individuals' familiarization with the test.