To reconstruct the electrophysiological activity of brain responses, source analysis is performed through the solution of the forward and inverse problems. The former contains a unique solution while the latter is ill-posed. In this regard, many algorithms have been suggested relying on different prior information for solving the inverse problem. Recently, neural networks have been used to deal with source analysis. However, their underlying training for inverse solutions is based on suboptimal forward modeling. In this work, we propose a CNN that is able to reconstruct EEG brain activity. To train our proposed CNN, a skull-conductivity calibrated and white matter anisotropic head model. Based on this model, we generate simulated EEG data and used them to train our CNN. We first evaluate the performance of our CNN using the simulated EEG data while a realistic application with somatosensory evoked potentials follows. From the results, we observed that the CCN correctly localized the P20/N20 component at the subject-specific Brodmann area 3b and it can potentially localize deeper sources. A comparison is also presented with well-known inverse solutions (single dipole scans and sLORETA) showing similar localization performance. Through these results, an emerging potential for real applications appears on the basis of realistic head modeling.