Despite the advances in image sensors, mainstream RGB sensors are still struggling from low quantum efficiency due to the low sensitivity of the Bayer color filter array. To address this issue, a sparse color sensor uses mostly panchromatic white pixels and a smaller percentage of sparse color pixels to provide better low-light photography performance than a conventional Bayer RGB sensor. However, due to the lack of a proper color reconstruction method, sparse color sensors have not been developed thus far. This study proposes a deep-learning-based method for sparse color reconstruction that can realize such a sparse color sensor. The proposed color reconstruction method consists of a novel two-stage deep model followed by an adversarial training technique to reduce visual artifacts in the reconstructed color image. In simulations and experiments, visual results and quantitative comparisons demonstrate that the proposed color reconstruction method can outperform existing methods. In addition, a prototype system was developed using a hybrid color-plus-mono camera system. Experiments using the prototype system reveal the feasibility of a very sparse color sensor in different lighting conditions.