Branching river networks are one of the most widespread and recognizable features of Earth's landscapes and have also been discovered elsewhere in the Solar System. But the mechanisms that create these patterns and control their spatial scales are poorly understood. Theories based on probability or optimality have proven useful, but do not explain how river networks develop over time through erosion and sediment transport. Here we show that branching at the uppermost reaches of river networks is rooted in two coupled instabilities: first, valleys widen at the expense of their smaller neighbours, and second, side slopes of the widening valleys become susceptible to channel incision. Each instability occurs at a critical ratio of the characteristic timescales for soil transport and channel incision. Measurements from two field sites demonstrate that our theory correctly predicts the size of the smallest valleys with tributaries. We also show that the dominant control on the scale of landscape dissection in these sites is the strength of channel incision, which correlates with aridity and rock weakness, rather than the strength of soil transport. These results imply that the fine-scale structure of branching river networks is an organized signature of erosional mechanics, not a consequence of random topology.