I present a general mathematical modeling framework that can provide a foundation for the study of sustainability in social- ecological systems (SESs). Using basic principles from feedback control and a sequence of specific models from bioeconomics and economic growth, I outline several mathematical and empirical challenges associated with the study of sustainability of SESs. These challenges are categorized into three classes: (1) the social choice of performance measures, (2) uncertainty, and (3) collective action. Finally, I present some opportunities for combining stylized dynamical systems models with empirical data on human behavior and biophysical systems to address practical challenges for the design of effective governance regimes (policy feedbacks) for highly uncertain natural resource systems.