Many human abilities rely on cognitive algorithms discovered by previous generations. Cultural accumulation of innovative algorithms is hard to explain because complex concepts are difficult to pass on. We found that selective social learning preserved rare discoveries of exceptional algorithms in a large experimental simulation of cultural evolution. Participants (N = 3450) faced a difficult sequential decision problem (sorting an unknown sequence of numbers) and transmitted solutions across 12 generations in 20 populations. Several known sorting algorithms were discovered. Complex algorithms persisted when participants could choose who to learn from but frequently became extinct in populations lacking this selection process, converging on highly transmissible lower-performance algorithms. These results provide experimental evidence for hypothesized links between sociality and cognitive function in humans.