The threat of obesity, diabetes, anorexia, and bulimia in our society today has motivated extensive research on dietary monitoring. Standard self-report methods such as 24-h recall and food frequency questionnaires are expensive, burdensome, and unreliable to handle the growing health crisis. Long-term activity monitoring in daily living is a promising approach to provide individuals with quantitative feedback that can encourage healthier habits. Although several studies have attempted automating dietary monitoring using wearable, handheld, smart-object, and environmental systems, it remains an open research problem. This paper aims to provide a comprehensive review of wearable and hand-held approaches from 2004 to 2016. Emphasis is placed on sensor types used, signal analysis and machine learning methods, as well as a benchmark of state-of-the art work in this field. Key issues, challenges, and gaps are highlighted to motivate future work toward development of effective, reliable, and robust dietary monitoring systems.