To ensure a correct interpretation of Proficiency Testing (PT) or External Quality Assessment (EQA) results, sample batches need to be homogeneous. The importance of homogeneity testing has been underlined by ISO 13528 and the IUPAC harmonized protocol for proficiency testing. They require that the batch heterogeneity should be smaller than 0.3σpt, with σpt the standard deviation for proficiency assessment according to ISO/IEC 17043:2010 and provide procedures for assessing the batch heterogeneity. In this study, a critical appraisal of these procedures is given. Firstly, an assessment is made of the influence of the criterion of 0.3σpt on laboratory evaluation. Secondly, the assessment procedure of the batch heterogeneity is evaluated, with an emphasis on the calculation and the evaluation of the measured batch standard deviation. It was found that the 0.3σpt criterion may be loosened without a detrimental effect on laboratory evaluation. In addition, it was shown that a flaw in the calculation procedure forces the batch standard deviation to be 0 with a probability of 21.4% for batch standard deviations of 0.3σpt and repeatability of 0.5σpt. The hypothesis test for accepting a batch exhibits a high probability of accepting the batch, even for batch standard deviations that are a multiple of 0.3σpt. To conclude, this study shows that the criteria for homogeneity check of PT items by ISO 13528 and the IUPAC protocol do not assure batch homogeneity. They may be a combination of a criterion that is too strict and a permissive assessment. Alternative assessment procedures should be followed.