PP01-01: Machine-Learning-Based Identification of Risk Factors for Immune-Related Adverse Events Using Clinical Trial Data
Poster Presenter
Brian Dreyfus
Senior Director
Bristol-Myers Squibb Company United States
Objectives
Evaluate feasibility of machine learning techniques to identify risk factors for immune-related adverse events associated with Immuno-oncology drugs to help manage patients’ safety in the clinical setting by enabling earlier diagnosis and treatment of adverse events leading to improved outcomes.
Method
We harmonized clinical trial data across 150+ studies and used standardized adverse event terms, medications, lab results, medical history notes, and comorbidities to train a deep neural network to predict occurrence of specific AEs. SHAP analysis was used on the model to interpret risk factors.
Results
A large integrated dataset of 159 oncology clinical trials was used to evaluate risk factors of cardiac events. There were approximately 48,000 subjects with over 2,000 relevant cardiac events. The select cardiac events for this analysis, were defined as having a diagnosis of one of the following 13 MedDRA Preferred Terms: pericarditis, ventricular arrhythmia, atrial flutter, myocardial infarction, myocarditis, supraventricular tachycardia, sinus tachycardia, atrial fibrillation, sinus bradycardia, pericardial effusion, ventricular tachycardia, restrictive cardiomyopathy and cardiac arrest. The cohort also included over 100,000 medical history terms, 16 million lab results and 11 million concomitant medication records. Over 9,000 unique medications were mapped to the Mechanism of Action (MoA) group using NIH guidelines resulting in 380 MoAs. Many model types were considered, but the top two chosen for further analysis and tuning were XGBoost and a deep neural network consisting of a set of embedding-layers for categorical features followed by 3 fully-connected layers. Models were trained to minimize loss on a binary classification target with weight applied to the minority class to compensate for class imbalance, and an 80/20% train/test split was applied. Model performance was evaluated using precision and recall metrics. For the deep neural network the precision was 19% and recall was 24% and for the XGBoost model the precision was 17% with a recall of 10%. The area under the PR-curve (PR-AUC) of the neural network model was 0.08 and the PR-AUC of the XGBoost was 0.056. The neural network model was selected for presentation of risk factors.
Conclusion
This study demonstrated that novel application of machine learning techniques can be applied to large integrated clinical trial datasets to augment signal detection and evaluation activities within pharmacovigilance to identify risk factors of adverse events. This study evaluated a composite endpoint of 13 different cardiac event preferred terms which yielded a sufficient sample size to conduct a machine learning analysis. This group of cardiac events were selected as some of the events (e.g. myocarditis) have a high risk of mortality and challenges with the differential diagnosis. Although, the AUCs of the models tested were not high, the features identified by the selected neural network model that were most important for the prediction were consistent with other analyses. The effort to integrate and confirm clinical trial data remains a challenge, but once completed the integrated dataset can provide the backbone for big data analyses. To further refine the research and improve the model performance, a follow-up analysis focused on a composite endpoint of severe cardiac events will be conducted.
