We conjecture that the leading two-derivative tree-level amplitudes for gluons and gravitons can be derived from gauge invariance together with mild assumptions on their singularity structure. Assuming locality (that the singularities are associated with the poles of cubic graphs), we prove that gauge invariance in just n-1 particles together with minimal power counting uniquely fixes the amplitude. Unitarity in the form of factorization then follows from locality and gauge invariance. We also give evidence for a stronger conjecture: assuming only that singularities occur when the sum of a subset of external momenta go on shell, we show in nontrivial examples that gauge invariance and power counting demand a graph structure for singularities. Thus, both locality and unitarity emerge from singularities and gauge invariance. Similar statements hold for theories of Goldstone bosons like the nonlinear sigma model and Dirac-Born-Infeld by replacing the condition of gauge invariance with an appropriate degree of vanishing in soft limits.