Regular nutrient intake monitoring in hospitalised patients plays a critical role in reducing the risk of disease-related malnutrition (DRM). Although several methods to estimate nutrient intake have been developed, there is still a clear demand for a more reliable and fully automated technique, as this could improve the data accuracy and reduce both the participant burden and the health costs. In this paper, we propose a novel system based on artificial intelligence to accurately estimate nutrient intake, by simply processing RGB depth image pairs captured before and after a meal consumption. For the development and evaluation of the system, a dedicated and new database of images and recipes of 322 meals was assembled, coupled to data annotation using innovative strategies. With this database, a system was developed that employed a novel multi-task neural network and an algorithm for 3D surface construction. This allowed sequential semantic food segmentation and estimation of the volume of the consumed food, and permitted fully automatic estimation of nutrient intake for each food type with a 15% estimation error.