Detrended fluctuation analysis (DFA) is one of the most widely used tools for the detection of long-range dependence in time series. Although DFA has found many interesting applications and has been shown to be one of the best performing detrending methods, its probabilistic foundations are still unclear. In this paper, we study probabilistic properties of DFA for Gaussian processes. Our main attention is paid to the distribution of the squared error sum of the detrended process. We use a probabilistic approach to derive general formulas for the expected value and the variance of the squared fluctuation function of DFA for Gaussian processes. We also get analytical results for the expected value of the squared fluctuation function for particular examples of Gaussian processes, such as Gaussian white noise, fractional Gaussian noise, ordinary Brownian motion, and fractional Brownian motion. Our analytical formulas are supported by numerical simulations. The results obtained can serve as a starting point for analyzing the statistical properties of DFA-based estimators for the fluctuation function and long-memory parameter.