Direct comparisons of neonatal outcomes at any level (unit, regional or international), require detailed validation and standardisation to ensure 'like for like' evaluation. Reported variation in neonatal performance may be either real or the result of one or more artefacts of the data collection. These issues need to be understood in order for an accurate interpretation to be made. Such artefacts are a particular feature of national data collection systems and can lead to serious misinterpretation. For example, very preterm deliveries have a major impact on neonatal mortality rates in developed countries with births before 33 weeks of gestation accounting for between 35% and 70% of neonatal deaths. Variation in the rate of very preterm delivery rates and differing practices regarding registration of these infants can have a major effect on the recorded neonatal mortality rate. At a more local level the validity of neonatal comparisons often depends upon whether the question being raised is appropriately matched to the data obtained to answer it. Problems arise when the question being addressed has been poorly framed or the data used to answer it has been inappropriately chosen. Comparisons using questions based on clearly defined standardised outcome measures and good quality prospective data collection are a much better way to proceed.