There is growing evidence from both experimental and non-experimental studies that fluctuating asymmetry does not consistently index stress or fitness. The widely held--yet poorly substantiated--belief that fluctuating asymmetry can act as a universal measure of developmental stability and predictor of stress-mediated changes in fitness, therefore staggers. Yet attempts to understand why the reported relationships between fluctuating asymmetry, stress and fitness are so heterogeneous--i.e. whether the associations are truly weak or non-existent or whether they become confounded during different stages of the analytical pathways remain surprisingly scarce. Hence, we attempt to disentangle these causes, by reviewing the various statistical and conceptual factors that are suspected to confound potential relationships between fluctuating asymmetry, stress and fitness. Two main categories of factors are discerned: those associated with the estimation of developmental stability through fluctuating asymmetry and those associated with the effects of genotype and environment on developmental stability. Next, we describe a series of statistical tools that have recently been developed to help reduce this noise. We argue that the current lack of a theoretical framework that predicts if and when relationships with developmental stability can be expected, urges for further theoretical and empirical research, such as on the genetic architecture of developmental stability in stressed populations. If the underlying developmental mechanisms are better understood, statistical patterns of asymmetry variation may become a biologically meaningful tool.