We consider far from equilibrium heat transport in chaotic billiard chains with noninteracting charged particles in the presence of nonuniform transverse magnetic field. If half of the chain is placed in a strong magnetic field, or if the strength of the magnetic field has a large gradient along the chain, heat current is shown to be asymmetric with respect to exchange of the temperatures of the heat baths. Thermal rectification factor can be arbitrarily large for sufficiently small temperature of one of the baths.