Using ultracold alkaline-earth atoms in optical lattices, we construct a quantum simulator for U(N) and SU(N) lattice gauge theories with fermionic matter based on quantum link models. These systems share qualitative features with QCD, including chiral symmetry breaking and restoration at nonzero temperature or baryon density. Unlike classical simulations, a quantum simulator does not suffer from sign problems and can address the corresponding chiral dynamics in real time.