Mandarine juice is one of the richest sources of β-cryptoxanthin and flavonoids, which have been positively associated with bone mineral density. Carotenoids are lipophilic isoprenoid compounds with a complex absorption process that can be affected by different factors. In this study, we have evaluated the effect of the food matrix on the in vitro bioaccessibility of carotenoids and phenolic compounds in a model milk-mandarine beverage (MMB). MMBs were formulated with mandarine juice and different dairy products to achieve three fat levels (0.2%, 1.7% and 3.2%) and three calcium levels (120, 310 and 500 mg Ca2+ per 100 ml). The bioaccessibility was evaluated using a harmonised in vitro digestion method. The results showed that the content of milk fat increased the bioaccessibility in vitro of phenolic compounds (p < 0.05), while a moderate fat level (1.7%) resulted in the highest bioaccessibility for bioactive carotenoids. On the other hand, calcium fortification at the highest level (500 mg Ca2+ per 100 mL) decreased the bioaccessibility of bioactive carotenoids from 76% to 43% (66% for the major β-cryptoxanthin) compared to the lower calcium fortification level (120 mg Ca2+ per 100 mL). The bioaccessibility of hesperidin, the main flavanone in mandarine juice, was significantly (p < 0.05) reduced in the MMB with the highest calcium level. The bioaccessibility of carotenoids and phenolic compounds is affected by fat and calcium levels. When formulating functional beverages, the impact of the formulation should be carefully considered to optimize the bioaccessibility of the bioactive compounds.