We demonstrate, what we believe to be, the first mathematical model of image formation in optical coherence tomography, based on Maxwell's equations, applicable to general three-dimensional samples. It is highly realistic and represents a significant advance on a previously developed model, which was applicable to two-dimensional samples only. The model employs an electromagnetic description of light, made possible by using the pseudospectral time-domain method for calculating the light scattered by the sample which is represented by a general refractive index distribution. We derive the key theoretical and computational advances required to develop this model. Two examples are given of image formation for which analytic comparisons may be calculated: point scatterers and finite sized spheres. We also provide a more realistic example of C-scan formation when imaging turbid media. We anticipate that this model will be important for various applications in OCT, such as image interpretation and the development of quantitative techniques.