Our perception of subjective difficulty in complex tasks, such as driving, is a judgment that is likely a result of dynamic interactions between distributed brain regions. In this paper, we investigate how neurophysiological markers associated with arousal state are informative of this perceived difficulty throughout a driving task. We do this by classifying subjective difficulty reports of subjects using set of features that include neural, autonomic, and eye behavioral markers. We subsequently assess the importance of these features in the classification. We find that though multiple EEG linked to cognitive control and, motor performance linked to classification of subjective difficulty, only pupil diameter, a measure of pupil-linked arousal, is strongly linked to both measured self-reported difficulty and actual task performance. We interpret our findings in the context of arousal pathways influencing performance and discuss their relevance to future brain-computer interface systems.