Associating polymers are a class of materials with widely tunable macroscopic properties. Here, we investigate telechelic poly(dimethylsiloxanes) of several molecular weights (MW) with different hydrogen bonding end groups. Besides the well-established increase of the glass transition temperature Tg with decreasing MW, Tg remains unchanged as the end group varies from NH2 over OH to COOH. For the latter system, a 2nd Tg is found which indicates a segregated phase. In contrast, rheological measurements reveal a qualitative difference in the viscoelastic response of NH2-terminated and COOH-terminated chains. Both systems show clear signs of end group association, but only the latter exhibits an extended rubbery plateau. All features observed in the rheology experiments have corresponding processes in the dielectric measurements. This provides insight into the underlying molecular mechanisms, and especially reveals that many end groups of the COOH-terminated chains phase segregate while a certain fraction forms binary associates and remains non-segregated. In contrast, the NH2-terminated systems form only binary associates increasing the effective chain length, whereas the COOH-terminated system consists of two types of associates forming a crosslinked network. Remarkably, a single species of end group forms two qualitatively different types of associates: transient bonds which allow stress release by a bond-partner exchange mechanism, and effectively permanent bonds formed by a phase segregated fraction of end groups which are stable on the timescale of the transient mechanism.