This paper proposes using power transfer matrix analysis to characterize the effects of Rayleigh backscattering and Fresnel reflection on WDM-PON systems. The modeling of a WDM-PON system can be carried out simply by matrix multiplication of the corresponding matrices for all the building blocks, where all possible guided backward lights and resonant configurations along the optical network can be accounted for. The total sum of all interferences affecting the bidirectional transmission that leads to an optical crosstalk-to-signal (C/S) ratio can be modeled as back-reflections through cascaded two-port networks for the downstream and upstream signals. This approach is simple, robust, efficient, and also accurate. Its accuracy is verified for simple system architectures and then applied to study more complicated cases. The results show its versatility to analyze a wide variety of bidirectional optical transmission systems.