To implant in the uterus, the mammalian embryo first specifies two cell lineages: the pluripotent inner cell mass that forms the fetus, and the outer trophectoderm layer that forms the placenta1. In many organisms, asymmetrically inherited fate determinants drive lineage specification2, but this is not thought to be the case during early mammalian development. Here we show that intermediate filaments assembled by keratins function as asymmetrically inherited fate determinants in the mammalian embryo. Unlike F-actin or microtubules, keratins are the first major components of the cytoskeleton that display prominent cell-to-cell variability, triggered by heterogeneities in the BAF chromatin-remodelling complex. Live-embryo imaging shows that keratins become asymmetrically inherited by outer daughter cells during cell division, where they stabilize the cortex to promote apical polarization and YAP-dependent expression of CDX2, thereby specifying the first trophectoderm cells of the embryo. Together, our data reveal a mechanism by which cell-to-cell heterogeneities that appear before the segregation of the trophectoderm and the inner cell mass influence lineage fate, via differential keratin regulation, and identify an early function for intermediate filaments in development.