Many stochastic systems in biology, physics and technology involve discrete time delays in the underlying equations of motion, stemming, e. g., from finite signal transmission times, or a time lag between signal detection and adaption of an apparatus. From a mathematical perspective, delayed systems represent a special class of non-Markovian processes with delta-peaked memory kernels. It is well established that delays can induce intriguing behaviour, such as spontaneous oscillations, or resonance phenomena resulting from the interplay between delay and noise. However, the thermodynamics of delayed stochastic systems is still widely unexplored. This is especially true for continuous systems governed by nonlinear forces, which are omnipresent in realistic situations. We here present an analytical approach for the net steady-state heat rate in classical overdamped systems subject to time-delayed feedback. We show that the feedback inevitably leads to a finite heat flow even for vanishingly small delay times, and detect the nontrivial interplay of noise and delay as the underlying reason. To illustrate this point, and to provide an understanding of the heat flow at small delay times below the velocity-relaxation timescale, we compare with the case of underdamped motion where the phenomenon of "entropy pumping" has already been established. Application to an exemplary (overdamped) bistable system reveals that the feedback induces heating as well as cooling regimes and leads to a maximum of the medium entropy production at coherence resonance conditions. These observations are, in principle, measurable in experiments involving colloidal suspensions.