In this work, we show how the structural features of photoactive azobenzene derivatives can influence the photoexcited state behavior and the yield of the trans/cis photoisomerization process. By combining high-resolution transient absorption experiments in the vis-NIR region and quantum chemistry calculations (TDDFT and RASPT2), we address the origin of the transient signals of three poly-substituted push-pull azobenzenes with an increasing strength of the intramolecular interactions stabilizing the planar trans isomer (absence of intramolecular H-bonds, methyl, and traditional H-bond, respectively, for 4-diethyl-4'-nitroazobenzene, Disperse Blue 366, and Disperse Blue 165) and a commercial red dye showing keto-enol tautomerism involving the azo group (Sudan Red G). Our results indicate that the intramolecular H-bonds can act as a "molecular lock" stabilizing the trans isomer and increasing the energy barrier along the photoreactive CNNC torsion coordinate, thus preventing photoisomerization in the Disperse Blue dyes. In contrast, the involvement of the azo group in keto-enol tautomerism can be employed as a strategy to change the nature of the lower excited state and remove the nonproductive symmetric CNN/NNC bending pathway typical of the azo group, thus favoring the productive torsional motion. Taken together, our results can provide guidelines for the structural design of azobenzene-based photoswitches with a tunable excited state behavior.