When assessing human health or ecosystem impacts of chemicals several calculation steps need to be addressed. Matrix algebra solving techniques are a useful approach to structure and solve the system of mass balance equations assessing chemical fate in environmental multimedia models. We suggest expanding this matrix approach towards a framework which includes the exposure, effect, and damage assessment for human health and ecosystems, also applicable to spatial modeling. Special emphasis is laid upon interpretation of the physical meaning of different elements within the matrices. The proposed framework provides several advantages such as simplified updating or extending of models to new impact pathways, possibility of covering various models within the same framework and transparency. Interpretation of intermediate and final results is facilitated, e.g., allowing for direct identification of dominating exposure pathways. Model comparability and evaluation is well supported, as the four matrices contain all intermediate results in a clear and interpretable way, independent from parameters, such as amount and place of emission. Multidisciplinary work is strongly facilitated enabling the linkage of different models from various disciplines together, since each of its modules defines a clear interface of intermediate results. This framework was reviewed by an independent expert panel within a UNEP/SETAC workshop, and adopted as starting-point for new advances in modeling environmental toxic releases within the UNEP/SETAC Life Cycle Initiative.