Enhancing Active Learning with Weak Supervision and Transfer Learning by Leveraging Information and Knowledge Sources
L. Rauch, D. Huseljic, and B. Sick. Workshop on Interactive Adaptive Learning (IAL), ECML PKDD, page 27--42. (2022)
One of the major limitations of deploying a machine learning
model is the availability of labeled training data and the resulting
expensive annotation process. Although active learning (AL) methods may
reduce the annotation cost by actively selecting the most-useful instances,
a costly human annotator usually provides the labels. Therefore, even
with AL, we still consider the annotation process to be time-consuming
and expensive. Besides human annotators, though, companies often have
a vast amount of information and knowledge sources available that can
generate low-cost labels (e.g., a black-box model) or improve the learning process
(e.g., a pre-trained model). We present a novel approach that
enhances AL with weak supervision (WS) and transfer learning (TL) to
reduce the annotation cost by leveraging these sources. Specifically, we
consider a black-box model like a rule-based system as an error-prone
and weakly-supervised annotator that inexpensively provides labels. We
estimate its performance with an annotator model to decide whether a
human annotation is required. Additionally, we utilize unlabeled internal
and external data by transferring knowledge from a pre-trained model
to the AL cycle. We sequentially investigate the impact of WS and TL
on annotation cost and model performance in an AL cycle through a use
case. Our evaluation shows that our approach can reduce annotation cost
by 51% while achieving nearly identical model performance compared to
a traditional AL approach.
