Binary patterns are used in fast Fourier single-pixel imaging (FSI) technology to increase the imaging speed at the expense of spatial resolution or image quality. In this Letter, we propose a method for optimizing the image quality-speed trade-off that is informed by physical principles and driven by data from simulations. To compensate for the quantization error induced by binary dithering, convolution kernels are proposed and optimized for both low and high spatial frequencies. The proposed method has been demonstrated to work in both simulation and experiments. Other single-pixel imaging (SPI) techniques may also benefit from this approach.