Although there are many known Mendelian genes linked to epileptic or developmental and epileptic encephalopathy (EE/DEE), its genetic architecture is not fully explained. Here, we address this incompleteness by analyzing exomes of 743 EE/DEE cases and 2366 controls. We observe that damaging ultra-rare variants (dURVs) unique to an individual are significantly overrepresented in EE/DEE, both in known EE/DEE genes and the other non-EE/DEE genes. Importantly, enrichment of dURVs in non-EE/DEE genes is significant, even in the subset of cases with diagnostic dURVs (P = 0.000215), suggesting oligogenic contribution of non-EE/DEE gene dURVs. Gene-based analysis identifies exome-wide significant (P = 2.04 × 10-6) enrichment of damaging de novo mutations in NF1, a gene primarily linked to neurofibromatosis, in infantile spasm. Together with accumulating evidence for roles of oligogenic or modifier variants in severe neurodevelopmental disorders, our results highlight genetic complexity in EE/DEE, and indicate that EE/DEE is not an aggregate of simple Mendelian disorders.