Nonlinear crystal frequency conversion imaging with direct detection by silicon-based detectors is an effective way to break through the limitations for existing near-infrared (NIR) detectors with expensive cost and high noise. In this paper, a broadband NIR detector imaging scheme based on the principle of nonlinear crystal frequency conversion (NCFCP) was proposed. A thin film of nonlinear crystal frequency conversion material (NCFCM) combined with a silicon-based detector was used to form a broadband NIR detector. The theoretically investigated energy transfer function was used as a guidance for experiment. Meanwhile, the relationship between the imaging effect and the energy transfer of the NCFCP-based compact broadband NIR detector in the NIR band was measured experimentally. The accuracy of the theoretical study had been verified by the measured transfer results.