It is not understood how changes in the genetic makeup of individuals alter the behavior of groups of animals. Here, we find that, even at early larval stages, zebrafish regulate their proximity and alignment with each other. Two simple visual responses, one that measures relative visual field occupancy and one that accounts for global visual motion, suffice to account for the group behavior that emerges. Mutations in genes known to affect social behavior in humans perturb these simple reflexes in individual larval zebrafish and change their emergent collective behaviors in the predicted fashion. Model simulations show that changes in these two responses in individual mutant animals predict well the distinctive collective patterns that emerge in a group. Hence, group behaviors reflect in part genetically defined primitive sensorimotor “motifs,” which are evident even in young larvae.