The Optimal Sample Size for Usability Testing, From the Manufacturer's Perspective: A Value-of-Information Approach

Abstract

Objectives: For medical devices, a usability assessment is mandatory for market access; the objective is to detect potentially harmful use errors that stem from the device's design. The manufacturer assesses the final version of the device and determines the risk-benefit ratio for remaining errors. Nevertheless, the decision rule currently used to determine the sample size for this testing has statistical limitations and the lack of a clear decision-making perspective.

Methods: As an alternative, we developed a value-of-information analysis from the medical device manufacturer's perspective. The consequences of use errors not detected during usability testing and the errors' probability of occurrence were embedded in a loss function. The value of further testing was assessed as a reduction in the expected loss for the manufacturer. The optimal sample size was determined using the expected net benefit of sampling (ENBS) (the difference between the value provided by new participants and the cost of their inclusion).

Results: The value-of-information approach was applied to a real usability test of a needle-free adrenaline autoinjector. The initial estimate (performed on the first n = 20 participants) gave an optimal sample size of 100 participants and an ENBS of €255 453. This estimation was updated iteratively as new participants were included. After the inclusion of 90 participants, the ENBS was null for any sample size; hence, the cost of adding more participants outweighed the expected value of information, and therefore, the study could be stopped.

Conclusions: On the basis of these results, our method seems to be highly suitable for sample size estimation in the usability testing of medical devices before market access.

Keywords: decision theory; early technology assessment; medical device; usability testing; value of information.