Obtaining substantial nonlinear effects at the single-photon level is a considerable challenge that holds great potential for quantum optical measurements and information processing. Of the progress that has been made in recent years one of the most promising methods is to scatter coherent light from quantum emitters, imprinting quantum correlations onto the photons. We report effective interactions between photons, controlled by a single semiconductor quantum dot that is weakly coupled to a monolithic cavity. We show that the nonlinearity of a transition modifies the counting statistics of a Poissonian beam, sorting the photons in number. This is used to create strong correlations between detection events and to create polarization-correlated photons from an uncorrelated stream using a single spin. These results pave the way for semiconductor optical switches operated by single quanta of light.