The viewing time method (VT) is an indirect measure of sexual interest in which participants are usually asked to rate the sexual attractiveness of target stimuli while their response latencies are unobtrusively measured. Higher response latencies for a certain group of depicted individuals indicate sexual interest in that group. Contrary to the general assumption that the picture content accounts for this effect by eliciting affect- or attention-based response delays, we hypothesized that the attractiveness rating task might be responsible for the VT effect. To test this hypothesis, we used two different tasks. Our heterosexual and homosexual male participants (N = 50) were instructed to rate the attractiveness or the sex of the depicted individuals. As expected, VT effects only emerged in attractiveness rating trials. Based on these findings, we conclude that VT effects are task dependent and are unlikely to be caused by affective or attentional processes (at least when participants are instructed to rate the attractiveness of target stimuli). We argue that rating tasks in VT measures cause participants to use affect independent response strategies. These response strategies seem to undermine stimulus-driven processes (like increased attention directed toward salient stimuli) which were thought to cause VT effects according to previous hypotheses.