We report the fundamental dependence of thermal diffusivity and electrical conductance on the diameter and defect level for vertically aligned single-walled carbon nanotube (SWCNT) forests. By synthesizing a series of SWCNT forests with continuous control of the diameter and defect level over a wide range while holding all other structures fixed, we found an inverse and mutually exclusive relationship between the thermal diffusivity and the electrical conductance. This relationship was explained by the differences in the fundamental mechanisms governing each property and the optimum required structures. We concluded that high thermal diffusivity and electrical conductance would be extremely difficult to simultaneously achieve by a single SWCNT forest structure within current chemical vapor deposition synthetic technology, and the "ideal" SWCNT forest structure would differ depending on application.