Inversion of the connectivity of amide groups in foldable azobenzene dyads with chiral side chains, which can self-assemble into toroids and nanotubes, significantly increases the thermal stability of these aggregates. The results can be explained by the geometrical difference of the amide groups. On the other hand, the side chain chirality is disregarded because the chiral centers are distant from the amide groups.