Cognitive functions are highly heritable and the impact of complex genetic interactions, though undoubtedly important, has received little investigation. Here we show in an animal model and in a human neuroimaging experiment a consistent non-linear interaction between two genes--catechol-O-methyl transferase (COMT) and dysbindin (dys; dystrobrevin-binding protein 1 (DTNBP1))--implicated through different mechanisms in cortical dopamine signaling and prefrontal cognitive function. In mice, we found that a single genetic mutation reducing expression of either COMT or DTNBP1 alone produced working memory advantages, while, in dramatic contrast, genetic reduction of both in the same mouse produced working memory deficits. We found evidence of the same non-linear genetic interaction in prefrontal cortical function in humans. In healthy volunteers (N=176) studied with functional magnetic resonance imaging during a working memory paradigm, individuals homozygous for the COMT rs4680 Met allele that reduces COMT enzyme activity showed a relatively more efficient prefrontal engagement. In contrast, we found that the same genotype was less efficient on the background of a dys haplotype associated with decreased DTNBP1 expression. These results illustrate that epistasis can be functionally multi-directional and non-linear and that a putatively beneficial allele in one epistastic context is a relatively deleterious one in another. These data also have important implications for single-locus association analyses of complex traits.