The drugs we use to cure our diseases can cause damage to the liver as it is the primary organ responsible for metabolism of environmental chemicals and drugs. To identify and eliminate potentially problematic drug candidates in the early stages of drug discovery, in silico techniques provide quick and practical solutions for toxicity determination. Deep learning has emerged as one of the solutions in recent years in the field of pharmaceutical chemistry. Generally, in the case of small data sets as used in toxicology, these data-hungry algorithms are prone to overfitting. We approach the problem from two sides. First, we use images of the three-dimensional conformations and benefit from convolutional neural networks which have fewer parameters than the standard deep neural networks with similar depth. Using images allows connecting various chemical features to the geometry of the compounds. Second, we employ the method COVER to up-sample the data set. It is used not only for increasing the size of the data set, but also for balancing the two classes, i.e., toxic and not toxic. The proof of concept is performed on the p53 end point from the Tox21 data set. The results, which are compatible with the winners of the data challenge, encouraged us to use our methods to predict liver toxicity. We use the most extensive publicly available liver toxicity data set by Mulliner et al. and obtain a sensitivity of 0.79 and a specificity of 0.52. These results demonstrate the applicability of image based toxicity prediction using deep neural networks.