The heartwood color of a major plantation tree Cryptomeria japonica shows high variability among clones and cultivars, and brighter heartwood has higher value in the usage of non-laminated wood such as in traditional construction, which makes heartwood color an important trait in breeding of this species. However, the genetic basis of the interactions between genetics and the environment on heartwood color has been understudied while these are necessary for effective breeding programs in multiple environmental condition. The objectives of the present study were to evaluate the effects of genetics and environments on heartwood color and how they interact in contrasting environments, and to identify genomic regions controlling heartwood color in C. japonica across multiple environments. Heartwood color in terms of L*a*b* color space and spectral reflectance was measured in common gardens established in three contrasting sites. Quantitative trait loci (QTL) that affect heartwood color were identified using previously constructed highly saturated linkage maps. Results found that heartwood color was largely genetically controlled, and genotype-by-environment interaction explained one-third of the total genetic variance of heartwood color. The effect of the environment was small compared to the effect of genetics, whereas environmental effects largely varied among heartwood color traits. QTL analysis identified a large number of QTLs with small to moderate effects (phenotypic variation explained of 6.6% on average). Some of these QTLs were stably expressed in multiple environments or had pleiotropic effects on heartwood color and moisture content. These results indicated that genetic variation in phenotypic plasticity plays an important role in regulating heartwood color and that the identified QTLs would maximize the breeding efficiency of heartwood color in C. japonica in heterogeneous environments.