We analyze a simple model of deterministic diffusion. The model consists of a one-dimensional array of scatterers with moving point particles. The particles move from one scatterer to the next according to a piecewise linear, expanding, deterministic map on unit intervals. The microscopic chaotic scattering process of the map can be changed by a control parameter. The macroscopic diffusion coefficient for the moving particles is well defined and depends upon the control parameter. We calculate the diffusion coefficent and the largest eigenmodes of the system by using Markov partitions and by solving the eigenvalue problems of respective topological transition matrices. For different boundary conditions we find that the largest eigenmodes of the map match the ones of the simple phenomenological diffusion equation. Our main result is that the diffusion coefficient exhibits a fractal structure as a function of the control parameter. We provide qualitative and quantitative arguments to explain features of this fractal structure.