Many organizations in the USA collect aquatic bioassessment data using different sampling and analysis methods, most of which have unknown performance in terms of data quality produced. Thus, the comparability of bioassessments produced by different organizations is often unknown, ultimately affecting our ability to make comprehensive assessments on large spatial scales. We evaluated a pilot approach for determining bioassessment performance using macroinvertebrate data obtained from several states in the Southeastern USA. Performance measures evaluated included precision, sensitivity, and responsiveness to a human disturbance gradient, defined in terms of a land disturbance index value for each site, combined with a value for specific conductance, and instream habitat quality. A key finding of this study is the need to harmonize ecoregional reference conditions among states so as to yield more comparable and consistent bioassessment results. Our approach was also capable of identifying potential areas for refinement such as reevaluation of less precise, sensitive, or responsive metrics that may result in suboptimal index performance. Higher performing bioassessments can yield information beyond "impaired" versus "unimpaired" condition. Acknowledging the limitations of this pilot study, we would recommend that performance evaluations use at least 50 sites, 10 of which are ecoregional reference sites. Efforts should be made to obtain data from the entire human disturbance gradient in an ecoregion to improve statistical confidence in performance measures. Having too few sites will result in an under-representation of certain parts of the disturbance gradient (e.g., too few poor quality sites), which may bias sensitivity and responsiveness estimates.