This Letter proposes a novel inverse halftoning algorithm that introduces a way of generating a pair of binary and continuous dictionaries optimized to a training image database composed of many pairs of halftoned patches and the corresponding continuous patches. The experiment results show that the two created binary and continuous dictionaries can be nicely used with the estimated sparse coefficients to represent an unknown continuous image with less noise and fine details from an input halftoned image.