We investigate the nonlinear light-matter interaction of a Bose-Einstein condensate trapped in an external periodic potential inside an optical cavity which is weakly coupled to vacuum radiation modes and driven by a transverse pump field. Based on a generalized Bose-Hubbard model which incorporates a single cavity mode, we include the collective backaction of the atoms on the cavity light field and determine the nonequilibrium quantum phases within the nonperturbative bosonic dynamical mean-field theory. With the system parameters adapted to recent experiments, we find a quantum phase transition from a normal phase to a self-organized superfluid phase, which is related to the Hepp-Lieb-Dicke superradiance phase transition. For even stronger pumping, a self-organized Mott insulator phase arises.