Recently, there has been a surge of interest in the measurement of mind wandering during ongoing tasks. The frequently used online thought-probing procedure (OTPP), in which individuals are probed on whether their thoughts are on-task or not while performing an ongoing task, has repeatedly been criticized, because variations in the frequency of thought probes and the order in which on-task and off-task thoughts are referred to have been shown to affect mind-wandering rates. Hitherto, it is unclear whether this susceptibility to measurement variation only affects mean response rates in probe-caught mind wandering or poses an actual threat to the validity of the OTPP, endangering the replicability and generalizability of study results. Here, we show in a sample of 177 students that variations of the frequency or framing of thought probes do not affect the validity of the OTPP. While we found that more frequent thought probing reduced the rate of probe-caught mind wandering, we did not replicate the effect that mind wandering is more likely to be reported when off-task thoughts are referred to first rather than second. Crucially, associations between probe-caught mind wandering and task performance, as well as associations between probe-caught mind wandering and covariates (trait mind wandering, reaction-time variability in the metronome-response task, and working-memory capacity) did not change with variations of the probing procedure. Therefore, it seems unlikely that the great heterogeneity in the way the OTPP is implemented across different studies endangers the replicability and generalizability of study results. Data and analysis code are available at https://osf.io/7w8bm/ .