Calculators are often unnecessary to solve routine problems, though they are convenient for offloading cognitively effortful processes. However, errors can arise if incorrect procedures are used or when users fail to monitor the output for keystroke mistakes. To investigate the conditions under which people's attention are captured by errant calculator outputs (i.e., from incorrectly chosen procedures or keystroke errors), we programmed an onscreen calculator to "lie" by changing the answers displayed on certain problems. We measured suspicion by tracking whether users explicitly reported suspicion, overrode calculator "lies", or re-checked their calculations after a "lie" was presented. In Study 1, we manipulated the concreteness of problem presentation and calculator delay between subjects to test how these affect suspicion towards "lies" (15% added to answers). We found that numeracy had no effect on whether people opted-in or out of using the calculator but did predict whether they would become suspicious. Very few people showed suspicion overall, however. For study 2, we increased the "lies" to 120% on certain answers and included questions with "conceptual lies" shown (e.g., a negative sign that should have been positive). We again found that numeracy had no effect on calculator usage, but, along with concrete formatting, did predict suspicion behavior. This was found regardless of "lie" type. For study 3, we reproduced these effects after offering students an incentive for good performance, which did raise their accuracy across the math problems overall but did not increase suspicion behavior. We conclude that framing problems within a concrete domain and being higher in numeracy increases the likelihood of spotting errant calculator outputs, regardless of incentive.