In addicts, associative memories related to the rewarding effects of drugs of abuse can evoke powerful craving and drug seeking urges, but effective treatment to suppress these memories is not available. Detailed insight into the neural circuitry that mediates expression of drug-associated memory is therefore of crucial importance. Substantial evidence from rodent models of addictive behavior points to the involvement of the ventromedial prefrontal cortex (vmPFC) in conditioned drug seeking, but specific knowledge of the temporal role of vmPFC pyramidal cells is lacking. To this end, we used an optogenetics approach to probe the involvement of vmPFC pyramidal cells in expression of a recent and remote conditioned cocaine memory. In mice, we expressed Channelrhodopsin-2 (ChR2) or Halorhodopsin (eNpHR3.0) in pyramidal cells of the vmPFC and studied the effect of activation or inhibition of these cells during expression of a cocaine-contextual memory on days 1-2 (recent) and ∼3 weeks (remote) after conditioning. Whereas optical activation of pyramidal cells facilitated extinction of remote memory, without affecting recent memory, inhibition of pyramidal cells acutely impaired recall of recent cocaine memory, without affecting recall of remote memory. In addition, we found that silencing pyramidal cells blocked extinction learning at the remote memory time-point. We provide causal evidence of a critical time-dependent switch in the contribution of vmPFC pyramidal cells to recall and extinction of cocaine-associated memory, indicating that the circuitry that controls expression of cocaine memories reorganizes over time.