Most covalent organic frameworks (COFs) to date are made from relatively small aromatic subunits, which can only absorb the high-energy part of the visible spectrum. We have developed near-infrared-absorbing low bandgap COFs by incorporating donor-acceptor-type isoindigo- and thienoisoindigo-based building blocks. The new materials are intensely colored solids with a high degree of long-range order and a pseudo-quadratic pore geometry. Growing the COF as a vertically oriented thin film allows for the construction of an ordered interdigitated heterojunction through infiltration with a complementary semiconductor. Applying a thienoisoindigo-COF:fullerene heterojunction as the photoactive component, we realized the first COF-based UV- to NIR-responsive photodetector. We found that the spectral response of the device is reversibly switchable between blue- and red-sensitive, and green- and NIR-responsive. To the best of our knowledge, this is the first time that such nearly complete inversion of spectral sensitivity of a photodetector has been achieved. This effect could lead to potential applications in information technology or spectral imaging.