Testing for interchangeability of imaging tests

Abstract

Rationale and objectives: New tests are typically assessed by estimating their technical and diagnostic performance through comparisons with a reference standard. A valid reference standard, however, is not always available and is not required for assessing the interchangeability of a new test with an existing one.

Materials and methods: To show interchangeability of a new test with an existing test, one compares the differences in diagnoses between the new and existing tests to differences between diagnoses made with the existing test on several occasions. We illustrate the test for interchangeability with two studies. In a transcatheter aortic valve replacement study, we test whether semiautomated analysis can be used interchangeably with manual reconstructions from three-dimensional computed tomography (CT) images. In patients with femoroacetabular impingement, we test whether magnetic resonance imaging (MRI) can replace CT to measure acetabular version.

Results: Although the semiautomated method agreed often with the manual measurement of aortic valve size (87.6%), interchanging the semiautomated method with manual measurements by an expert would lead to a 1.7%-12.2% increase in the frequency of disagreement. Interchanging MRI for CT to measure acetabular version would lead to differences in angle measurements of 2.0° to 3.1° in excess of the differences we would expect to see with CT alone.

Conclusions: Testing for agreement or correlation between a new and an existing test is not sufficient evidence of the performance of a new test. A formal evaluation of interchangeability can be conducted in the absence of a reference standard.

Keywords: Reference standard; accuracy; agreement; correlation; interchangeability.