Seafood derived from wild fish as well as farmed fish has always been an important source of protein in the human diet. On a global scale, fish and fish products are the most important source of protein and it is estimated that more than 30% of fish for human consumption comes from aquaculture. The first part of this paper outlines the hazards and challenges associated with handling fish during farming and capture. The authors describe infectious agents that cause disease in fish as well as humans, zoonotic agents, intoxications due to bacteria and allergies caused by the consumption of fish. Although only a few infectious agents in fish are able to infect humans, some exceptions exist that may result in fatalities. However, the greatest risk to human health is due to the consumption of raw or insufficiently processed fish and fish products. The second part of the paper considers environmental contaminants in seafood that may pose a risk to human health, such as medicinal products and residues associated with aquaculture, persistent lipophilic organic compounds and metals (methyl-mercury, organotin). The authors include an updated overview of the various factors associated with farmed and captured fish that may cause risks to human health after consumption. Moreover, they discuss the challenges (in the widest sense) associated with handling fish during capture and farming, as well as those encountered during processing.