Recent years have seen dramatic developments in the technology of intense pulsed light sources in the THz frequency range. Since many dipole-active excitations in solids and molecules also lie in this range, there is now a tremendous potential to use these light sources to study linear and nonlinear dynamics in such systems. While several experimental investigations of THz-driven dynamics in solid-state systems have demonstrated a variety of interesting linear and nonlinear phenomena, comparatively few efforts have been made to drive analogous dynamics in molecular systems. In the present Perspective article, we discuss the similarities and differences between THz-driven dynamics in solid-state and molecular systems on both conceptual and practical levels. We also discuss the experimental parameters needed for these types of experiments and thereby provide design criteria for a further development of this new research branch. Finally, we present a few recent examples to illustrate the rich physics that may be learned from nonlinear THz excitations of phonons in solids as well as inter-molecular vibrations in liquid and gas-phase systems.