Ultrashort pulses at infrared wavelengths are advantageous when studying light-matter interaction. For the spectral region around 2 µm, multi-stage parametric amplification is the most common method to reach higher pulse energies. Yet it has been a key challenge for such systems to deliver waveform-stable pulses without active stabilization and synchronization systems. Here, we present a different approach for the generation of infrared pulses centered at 1.8 µm with watt-level average power utilizing only a single nonlinear crystal. Our laser system relies on a well-established Yb:YAG thin-disk technology at 1.03 µm wavelength combined with a hybrid two-stage broadening scheme. We show the high-power downconversion process via intra-pulse difference frequency generation, which leads to excellent passive stability of the carrier envelope phase below 20 mrad-comparable to modern oscillators. It also provides simple control over the central wavelength within a broad spectral range. The developed infrared source is employed to generate a multi-octave continuum from 500 nm to 2.5 µm opening the path toward sub-cycle pulse synthesis with extreme waveform stability.