We propose a broad-spectrum diffractive deep neural network (BS-D2NN) framework, which incorporates multiwavelength channels of input lightfields and performs a parallel phase-only modulation using a layered passive mask architecture. A complementary multichannel base learner cluster is formed in a homogeneous ensemble framework based on the diffractive dispersion during lightwave modulation. In addition, both the optical sum operation and the hybrid (optical-electronic) maxout operation are performed for motivating the BS-D2NN to learn and construct a mapping between input lightfields and truth labels under heterochromatic ambient lighting. The BS-D2NN can be trained using deep learning algorithms to perform a kind of wavelength-insensitive high-accuracy object classification.