A technique to improve the combustion performance of a hybrid rocket engine using a novel fuel grain structure is presented. This technique utilizes the different regression rates of acrylonitrile butadiene styrene and paraffin-based fuels, which increase the exchanges of both matter and energy by swirl flow and recirculation zones formed at the grooves between the adjacent vanes. The centrifugal casting technique is used to cast the paraffin-based fuel into an acrylonitrile butadiene styrene substrate made by three-dimensional printing. Using oxygen as the oxidizer, a series of tests were conducted to investigate the combustion performance of the novel fuel grain. In comparison to paraffin-based fuel grains, the fuel grain with a nested helical structure, which can be maintained throughout the combustion process, showed significant improvement in the regression rate and great potential in improvement of combustion efficiency.