The consumption of mixed nuts is a healthy dietary strategy to reduce the risk of cardiovascular disease and has a prebiotic effect on the gut microbiota. However, there is a lack of basic research based on mixed nut formulation. This study established a new method for optimizing mixed nut formulations using the Nutrient Rich Food (NRF) index model. Nutrient indices were adjusted by combining 10 and 8 encouraging nutrients and 3 limiting nutrients of nuts and dried fruits, respectively. The optimized mixed nut formulation had the highest total NRF and the lowest energy, which was achieved by applying linear programming. The effect of an optimized mixed nut formulation on insulin resistance and gut microbiota was investigated in an animal model of metabolic disorders caused by a high-fat diet. Male C57BL/6J mice (n = 12 per group) were fed a low-fat diet, a high-fat diet (HFD), HFD with a supplemented classical randomized controlled trial mixed nut formula (MN1), a commercially available mixed nut formula (MN2), a high-nutrient density mixed nut formula (MN3), or ellagic acid (positive control). MN3 treatment decreased total plasma cholesterol, homeostasis model assessment-insulin resistance index, high sensitivity C-reactive protein, and zonulin levels, strengthened the intestinal barrier, and significantly altered the β-diversity of the intestinal microbiota as compared to the HFD group. These effects of MN3 were superior to MN1 and MN2. In conclusion, MN3 had the highest nutrient density and improved insulin resistance in low-grade inflammation via gut microbiota remodeling.