T-01: Use of Adverse Event Data to Develop an Artificial Intelligence Application for Assessing Seriousness

Poster Presenter

Bruno Assuncao

Associate Director Pharmacovigilance Innovation
Celgene
United States

Objectives

Adverse event (AE) seriousness is a factor in defining reporting timelines and is often performed manually by pharmacovigilance experts. We used AE reports, source documents, and associated metadata to develop a machine learning methodology for automatically identifying adverse event seriousness.

Method

Using a stratified random sample of AE reports and their associated metadata, we developed three artificial intelligence (AI) algorithms for addressing identification of adverse event seriousness. Models were evaluated for accuracy and/or F1 score based on sensitivity, specificity and precision.

Results

We developed three artificial intelligence (AI) algorithms for addressing identification of adverse event seriousness: 1) a binary classifier for determining document-level seriousness, 2) a multi-classifier for determining seriousness categorization at the adverse-event level, and 3) Case-level category annotator for supporting human text-based review to determine seriousness categorization at the AE level. Classifiers were evaluated for accuracy and the annotator for F1 score based on sensitivity (recall), specificity, and precision (as applicable) against a ground truth of manual review by pharmacovigilance experts. The seriousness classifier achieved an accuracy of 83.0% in post marketing reports, 95.8% in solicited reports, and 84.7% in medical literature reports. F1 scores for seriousness categorization were 78.9 for death, 77.7 for hospitalization, and 75.5 for important medical events. The seriousness annotator achieved an F1 score of 49.0 in post marketing reports, 89.9 in solicited reports, and 71.1 in medical literature reports.

Conclusion

Volume, complexity, and time constraints of adverse event reporting is overwhelming the PV workforce. New solutions are needed to support these activities to meet global regulatory timelines. We developed several machine learning approaches to support the correct identification and classification of seriousness, a key factor in adverse reporting, in various document types. Our deep learning models were trained using an extensive data set that captured deep institutional pharmacovigilance practitioner knowledge. The results of this study indicate that a neural network approach can provide an accurate and scalable solution for potentially augmenting pharmacovigilance practitioner determination of adverse event seriousness in spontaneous, solicited, and medical literature reports.