A crucial question in mammalian development is how cells of the early embryo differentiate into distinct cell types. The first decision is taken when cells undertake waves of asymmetric division that generate one daughter on the inside and one on the outside of the embryo. After this division, some cells on the inside remain pluripotent and give rise to the epiblast, and hence the future body, whereas others develop into the primitive endoderm, an extraembryonic tissue. How the fate of these inside cells is decided is unknown: Is the process random, or is it related to their developmental origins? To address this question, we traced all cells by live-cell imaging in intact, unmanipulated embryos until the epiblast and primitive endoderm became distinct. This analysis revealed that inner cell mass (ICM) cells have unrestricted developmental potential. However, cells internalized by the first wave of asymmetric divisions are biased toward forming pluripotent epiblast, whereas cells internalized in the next two waves of divisions are strongly biased toward forming primitive endoderm. Moreover, we show that cells internalized by the second wave up-regulate expression of Gata6 and Sox17, and changing the expression of these genes determines whether the cells become primitive endoderm. Finally, with our ability to determine the origin of cells, we find that inside cells that are mispositioned when they are born can sort into the correct layer. In conclusion, we propose a model in which the timing of cell internalization, cell position, and cell sorting combine to determine distinct lineages of the preimplantation mouse embryo.