Evaluation Strategies for Understanding Experiences With Virtual Care in Canada: Mixed Methods Study

Abstract

Background: Virtual care was rapidly integrated into pediatric health services during the COVID-19 pandemic. While virtual care offers many benefits, it is necessary to better understand the experiences of those who receive, deliver, and coordinate virtual care in order to support sustainable, high-quality, and patient-centered health care. To date, methods implemented to evaluate users' experiences of virtual care have been highly variable, making comparison and data synthesis difficult.

Objective: This study aims to describe evaluation strategies currently used to understand personal experiences with pediatric virtual care in Canada.

Methods: In this mixed methods environmental scan, we first distributed a web-based questionnaire to clinical, research, and operational leaders delivering and evaluating pediatric virtual care in Canada. The questionnaire collected information about how experiences with virtual care have been or are currently being evaluated and whether these evaluations included the perspectives of children or youth, families, providers, or support staff. Second, respondents were asked to share the questions they used in their evaluations, and a content analysis was performed to identify common question categories. Third, we conducted semistructured interviews to further explore our respondents' evaluation experiences across 4 domains-evaluation approaches, distribution methods, response rates, and lessons learned-and interest in a core set of questions for future evaluations.

Results: There were 72 respondents to the web-based questionnaire; among those who had conducted an evaluation, we identified 15 unique evaluations, and 14 of those provided a copy of the tools used to evaluate virtual care. These evaluations measured the virtual care experiences of parents or caregivers (n=15, 100%), children or youth (n=11, 73%), health care providers (n=11, 73%), and support staff (n=4, 27%). The most common data collection method used was electronic questionnaires distributed by email. Two respondents used validated tools; the remainder modified existing tools or developed new tools. Content analysis of the 14 submitted questionnaires revealed that the most common questions were about overall participant satisfaction, the comparison of virtual care to in-person care, and whether participants would choose virtual care options in the future. Interview findings indicate respondents frequently relied on methods used by peers and that a standardized, core set of questions to evaluate experiences with virtual care would be helpful to improve evaluation practices and support pediatric health care delivery.

Conclusions: At our institution and elsewhere in Canada, experiences with pediatric virtual care have been evaluated using a variety of methods. A more consistent evaluation approach using standardized tools may enable more regular comparisons of experiences with virtual care and the synthesis of findings across health care settings. In turn, this may better inform our approach to virtual care, improve its integration into health systems, and facilitate sustainable, high-quality, patient-centered care.

Keywords: environmental scan; experience; interviews; pediatrics; telemedicine; virtual care.