We present a novel technique for computing the free energy differences between two chromophore "isomers" hosted in a molecular environment (a generalized solvent). Such an environment may range from a relatively rigid protein cavity to a flexible solvent environment. The technique is characterized by the application of the previously reported "average electrostatic solvent configuration" method, and it is based on the idea of using the free energy perturbation theory along with a chromophore annihilation procedure in thermodynamic cycle calculations. The method is benchmarked by computing the ground-state room-temperature relative stabilities between (i) the cis and trans isomers of prototypal animal and microbial rhodopsins and (ii) the analogue isomers of a rhodopsin-like light-driven molecular switch in methanol. Furthermore, we show that the same technology can be used to estimate the activation free energy for the thermal isomerization of systems i-ii by replacing one isomer with a transition state. The results show that the computed relative stability and isomerization barrier magnitudes for the selected systems are in line with the available experimental observation in spite of their widely diverse complexity.