Robust importance sampling for error estimation in the context of optimal Bayesian transfer learning

Omar Maddouri; Xiaoning Qian; Francis J Alexander; Edward R Dougherty; Byung-Jun Yoon

doi:10.1016/j.patter.2021.100428

Robust importance sampling for error estimation in the context of optimal Bayesian transfer learning

Patterns (N Y). 2022 Jan 25;3(3):100428. doi: 10.1016/j.patter.2021.100428. eCollection 2022 Mar 11.

Authors

Omar Maddouri¹, Xiaoning Qian^{1

2}, Francis J Alexander², Edward R Dougherty¹, Byung-Jun Yoon^{1

2}

Affiliations

¹ Department of Electrical and Computer Engineering, Texas A&M University, College Station, TX 77843, USA.
² Computational Science Initiative, Brookhaven National Laboratory, Upton, NY 11973, USA.

Abstract

Classification has been a major task for building intelligent systems because it enables decision-making under uncertainty. Classifier design aims at building models from training data for representing feature-label distributions-either explicitly or implicitly. In many scientific or clinical settings, training data are typically limited, which impedes the design and evaluation of accurate classifiers. Atlhough transfer learning can improve the learning in target domains by incorporating data from relevant source domains, it has received little attention for performance assessment, notably in error estimation. Here, we investigate knowledge transferability in the context of classification error estimation within a Bayesian paradigm. We introduce a class of Bayesian minimum mean-square error estimators for optimal Bayesian transfer learning, which enables rigorous evaluation of classification error under uncertainty in small-sample settings. Using Monte Carlo importance sampling, we illustrate the outstanding performance of the proposed estimator for a broad family of classifiers that span diverse learning capabilities.

Keywords: Bayesian error estimator; OBTL; classification; error estimation; importance sampling; model uncertainty; optimal Bayesian transfer learning; transfer learning.