Holographic optical element (HOE) can be used in many areas in optics due to its characteristics of thin structure, flexible wavefront reconstruction/control ability and angular/wavelength selectivity. In this paper, we propose a design method of off-axis reflective imaging systems based on freeform HOEs, which are fabricated by freeform wavefronts. The freeform HOEs offer many degrees of design freedom and can correct the aberrations in nonsymmetric imaging systems. The initial imaging system with freeform HOEs is generated using a point-by-point design approach, and is used for the preliminary design of the imaging system and the freeform recording system of each HOE. Then a joint optimization is conducted for all the systems, simultaneously considering the imaging performance, the diffraction efficiency, the system constraints and fabrication to get the final design results. To validate the feasibility and effectiveness of the proposed method, an off-axis reflective head-up display system with good performance based on freeform HOEs is designed and fabricated. Detailed procedures of the design and development process of the prototype are demonstrated.