The light switchable thermal conductivity displayed by some polymers makes them promising for applications like data storage, temperature regulation and light switchable devices. In this study, the mechanism of thermal conductivity switching in poly[6-(4-phenyldiazenyl phenoxy)hexyl metharylate] is studied using molecular dynamics (MD) simulations. The π-π stacking and amorphous polymer structures are specifically prepared through different simulation procedures, and the thermal conductivity of these structures is calculated. It is found that due to the π-π stacking structure, the thermal conductivity along the side-chain direction can change by 30-50% (from 0.34 to 0.51 W m-1 K-1). Through heat flux decomposition, it is found that the thermal conductivity change is dominated by the contribution from bonding interactions. This is because π-π stacking, which enforces the trans conformation, extends the side-chains of azobenzene polymers, making thermal transport in the side-chain direction more efficient. Along the polymer main-chain direction, the thermal conductivity is not affected by the π-π stacking of the side chains. This mechanism may be generalized to other types of polymers with azobenzene side-chains, which will develop a class of photo-responsive polymers.