We propose a high-speed image-reconstruction algorithm for a spatially multiplexed image that is obtained by spatial frequency-division multiplexing. The algorithm utilizes smoothing and does not require any Fourier transform (FT) or iterative procedure to extract the desired information selectively from a single image. Numerical and experimental results show its validity and color holographic imaging ability. Calculation time of the proposed is less than a tenth of that of the FT method when using a central processing unit and an image sensor with four megapixels. Furthermore, throughput that is three times that of the FT method can be achieved in the case of using a graphics processing unit.