We consider the linear and quadratic higher-order terms associated with the response of the statistical properties of a dynamical system to suitable small perturbations. These terms are related to the first and second derivative of the stationary measure with respect to the changes in the system itself, expressing how the statistical properties of the system vary under the perturbation. We show a general framework in which one can obtain rigorous convergence and formulas for these two terms. The framework is flexible enough to be applied both to deterministic and random systems. We give examples of such an application computing linear and quadratic response for Arnold maps with additive noise and deterministic expanding maps.