Cells remember their identities, in part, by using epigenetic marks-chemical modifications placed along the genome. How can mark patterns remain stable over cell generations despite their constant erosion by replication and other processes? We developed a theoretical model that reveals that three-dimensional (3D) genome organization can stabilize epigenetic memory as long as (i) there is a large density difference between chromatin compartments, (ii) modifying "reader-writer" enzymes spread marks in three dimensions, and (iii) the enzymes are limited in abundance relative to their histone substrates. Analogous to an associative memory that encodes memory in neuronal connectivity, mark patterns are encoded in a 3D network of chromosomal contacts. Our model provides a unified account of diverse observations and reveals a key role of 3D genome organization in epigenetic memory.