In some cases the state of a quantum system with a large number of subsystems can be approximated efficiently by the density-matrix renormalization group, which makes use of redundancies in the description of the state. Here we show that the achievable efficiency can be much better when performing density-matrix renormalization group calculations in the Heisenberg picture, as only the observable of interest but not the entire state is considered. In some nontrivial cases, this approach can even be exact for finite bond dimensions.