It is widely recognized that balancing excitation and inhibition is important in the nervous system. When such a balance is sought by global strategies, few modes remain poised close to instability, and all other modes are strongly stable. Here we present a simple abstract model in which this balance is sought locally by units following "anti-Hebbian" evolution: all degrees of freedom achieve a close balance of excitation and inhibition and become "critical" in the dynamical sense. At long time scales, a complex "breakout" dynamics ensues in which different modes of the system oscillate between prominence and extinction; the model develops various long-tailed statistical behaviors and may become self-organized critical.