As a component of diagnostic test evaluation, the estimation of repeatability and reproducibility of an assay is necessary to assess the robustness and the transferability of the method among laboratories. Respectively defined as the agreement within and between laboratories, repeatability and reproducibility of a qualitative diagnostic test are traditionally reported using observed proportion of agreement or Kappa values. Applied to a recently designed RT-PCR assay for the detection of infectious salmon anaemia virus, repeatability only within a national reference laboratory and reproducibility with two additional independent regional laboratories were investigated. Homogenization of fish kidney tissue was conducted to potentially provide more uniform submission material, and to assess the effect of homogenization on laboratory comparability. Comparison of agreement between non-homogenized and homogenized tissue samples revealed different patterns of test results and unexpected alterations of agreement due to homogenization. This observation may be explained by cross-contamination of some samples during the homogenization process. One of the laboratories was in clear disagreement with the two others and impacted the overall reproducibility of the assay. Agreement levels were visually described using a novel tree-shape representation inspired from phylogenetic studies. The resulting phylogram illustrated the proximity of test findings between repeated samples within a laboratory and between laboratories, and facilitated the interpretation of the agreement levels.