In multispectral color imaging, there is a demand to select a reduced number of optimal imaging channels to simultaneously speed up the image acquisition process and keep reflectance reconstruction accuracy. In this paper, the channel selection problem is cast as the binary optimization problem, and is consequently solved using a novel binary differential evolution (DE) algorithm. In the proposed algorithm, we define the mutation operation using a differential table of swapping pairs, and deduce the trial solutions using neighboring self-crossover. In this manner, the binary DE algorithm can well adapt to the channel selection problem. The proposed algorithm is evaluated on the multispectral color imaging system on both synthetic and real data sets. It is verified that high color accuracy is achievable by only using a reduced number of channels using the proposed method. In addition, as binary DE is a global optimization algorithm in nature, it performs better than the traditional sequential channel selection algorithm.