A framework to identify ethical concerns with ML-guided care workflows: a case study of mortality prediction to guide advance care planning

Diana Cagliero; Natalie Deuitch; Nigam Shah; Chris Feudtner; Danton Char

doi:10.1093/jamia/ocad022

A framework to identify ethical concerns with ML-guided care workflows: a case study of mortality prediction to guide advance care planning

J Am Med Inform Assoc. 2023 Apr 19;30(5):819-827. doi: 10.1093/jamia/ocad022.

Authors

Diana Cagliero¹, Natalie Deuitch^{2

3}, Nigam Shah⁴, Chris Feudtner^{5

6}, Danton Char^{7

8}

Affiliations

¹ Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada.
² Department of Genetics, Stanford University School of Medicine, Stanford, California, USA.
³ National Institutes of Health, National Human Genome Research Institute, Bethesda, Maryland, USA.
⁴ Center for Biomedical Informatics Research, Stanford University School of Medicine, Palo Alto, California, USA.
⁵ The Department of Medical Ethics, The Children's Hospital of Philadelphia, Philadelphia, Pennsylvania, USA.
⁶ Departments of Pediatrics, Medical Ethics and Healthcare Policy, The Perelman School of Medicine, The University of Pennsylvania, Philadelphia, Pennsylvania, USA.
⁷ Division of Pediatric Cardiac Anesthesia, Department of Anesthesiology, Stanford University School of Medicine, Stanford, California, USA.
⁸ Center for Biomedical Ethics, Stanford University School of Medicine, Stanford, California, USA.

Abstract

Objective: Identifying ethical concerns with ML applications to healthcare (ML-HCA) before problems arise is now a stated goal of ML design oversight groups and regulatory agencies. Lack of accepted standard methodology for ethical analysis, however, presents challenges. In this case study, we evaluate use of a stakeholder "values-collision" approach to identify consequential ethical challenges associated with an ML-HCA for advanced care planning (ACP). Identification of ethical challenges could guide revision and improvement of the ML-HCA.

Materials and methods: We conducted semistructured interviews of the designers, clinician-users, affiliated administrators, and patients, and inductive qualitative analysis of transcribed interviews using modified grounded theory.

Results: Seventeen stakeholders were interviewed. Five "values-collisions"-where stakeholders disagreed about decisions with ethical implications-were identified: (1) end-of-life workflow and how model output is introduced; (2) which stakeholders receive predictions; (3) benefit-harm trade-offs; (4) whether the ML design team has a fiduciary relationship to patients and clinicians; and, (5) how and if to protect early deployment research from external pressures, like news scrutiny, before research is completed.

Discussion: From these findings, the ML design team prioritized: (1) alternative workflow implementation strategies; (2) clarification that prediction was only evaluated for ACP need, not other mortality-related ends; and (3) shielding research from scrutiny until endpoint driven studies were completed.

Conclusion: In this case study, our ethical analysis of this ML-HCA for ACP was able to identify multiple sites of intrastakeholder disagreement that mark areas of ethical and value tension. These findings provided a useful initial ethical screening.

Keywords: artificial intelligence; clinical; end-of-life care; ethics; machine learning; palliative care.

Publication types

Research Support, Non-U.S. Gov't

MeSH terms

Advance Care Planning*
Ethnicity
Humans
Qualitative Research
Workflow