Quantitative isotropic diffusion MRI and voxel-based analysis of the apparent diffusion coefficient (ADC) changes have been demonstrated to be able to accurately predict early response of brain tumors to therapy. The ADC value changes measured during pre- and posttherapy interval are closely correlated to treatment response. This work was demonstrated using a voxel-based analysis of ADC change during therapy in the brains of both rats and humans, following rigidly registering pre- and post-therapeutic ADC MRI exams. The primary goal of this paper is to extend this voxel-by-voxel analysis to assess therapeutic response in breast cancer. Nonlinear registration (with higher degrees of freedom) between the pre- and post-treatment exams is needed to ensure that the corresponding voxels actually contain similar cellular partial contributions due to soft tissue deformations in the breast and compartmental tumor changes during treatment as well. With limited data sets, we have observed the correlation between changes of ADC values and treatment response also exists in breast cancers. With diffusion scans acquired at three different timepoints (pre-treatment, early post-treatment and late post-treatment), we have also shown that ADC changes across responders within 5 weeks are a function of time interval after the initiation of treatment. Comparison of the experimental results with pathology shows that ADC changes can be used to evaluate early response of breast cancer treatment.