This paper introduces a novel framework for diffusion tensor combination, which can be used for tensor averaging, clustering, interpolation, and regularization. The framework is based on the physical interpretation of the tensors as the covariance matrices of Gaussian probability distributions. The symmetric Kullback-Leibler divergence provides a natural distance measure on these distributions, which leads to a closed-form expression for the distance between any two diffusion tensors, as well as for the weighted average of an arbitrary number of tensors. We illustrate the application of our technique in four different scenarios: (a) to combine tensor data from multiple subjects and generate population atlases from ten young and ten older subjects, (b) to perform k-means clustering and generate a compact Gaussian mixture of multiple tensors, (c) to interpolate between tensors, and (d) to regularize (i.e., smooth) noisy tensor data. For boundary-preserving regularization, we also propose a non-linear two-stage smoothing algorithm that can be considered remotely similar to a median filter.