We develop a novel radiotherapy plan comparison index, critical organ scoring index (COSI), which is a measure of both target coverage and critical organ overdose. COSI is defined as COSI=1-(V(OAR)>tol/TC), where V(OAR)>tol is the fraction of volume of organ at risk receiving more than tolerance dose, and TC is the target coverage, VT,PI/VT, where VT,PI is the target volume receiving at a least prescription dose and VT is the total target volume. COSI approaches unity when the critical structure is completely spared and the target coverage is unity. We propose a two-dimensional, graphical representation of COSI versus conformity index (CI), where CI is a measure of a normal tissue overdose. We show that this 2D representation is a reliable, visual quantitative tool for evaluating competing plans. We generate COSI-CI plots for three sites: head and neck, cavernous sinus, and pancreas, and evaluate competing non-coplanar 3D and IMRT treatment plans. For all three sites this novel 2D representation assisted the physician in choosing the optimal plan, both in terms of target coverage and in terms of critical organ sparing. We verified each choice by analysing individual DVHs and isodose lines. Comparing our results to the widely used conformation number, we found that in all cases where there were discrepancies in the choice of the best treatment plan, the COSI-CI choice was considered the correct one, in several cases indicating that a non-coplanar 3D plan was superior to the IMRT plans. The choice of plan was quick, simple and accurate using the new graphical representation.