Inorganic nanocrystals have a variety of applications in medicine. They may serve as contrast agents, therapeutics, and for in vitro diagnostics. Frequently, the synthesis route yields hydrophobically capped nanocrystals, which necessitates their subsequent coating to render a water-soluble and biocompatible probe. Biocompatibility is crucial for cellular imaging applications, which require large quantities of diagnostically active nanoparticles to be loaded into cells. We have previously reported the design and synthesis of a fluorescent and magnetic resonance imaging-detectable core-shell nanoparticle that encapsulates hydrophobically coated iron oxide nanocrystals. The core of soybean oil and iron oxide is covered by a shell mixture of phospholipids, some of which contained polyethylene glycol. Despite the biocompatibility of these components, we hypothesize that we can improve this formulation with respect to in vitro toxicity. To this aim, we measured the effect of six different core compositions on nanoparticle structure, cell labeling efficacy, and cell viability, as well as cell tracking potential. We methodically investigated the causes of toxicity and conclude that, even when combining biocompatible materials, the resulting formulation is not guaranteed to be biocompatible.