P111: Remote Monitoring of Adverse Event Under-Reporting in Clinical Trials: Implications for Critical to Quality Issue Detection
Poster Presenter
Jennifer Emerson
Head Quality Analytics & Risk Management
Boehringer Ingelheim Germany
Objectives
The objective was to evaluate Bootstrap Resampling (BSR), a Machine Learning technique, under different conditions to understand the consistency and accuracy of the BSR method for the detection of Adverse Event under-reporting and how early in a clinical trial first identification would be possible.
Method
The authors used simulated clinical trial (SCT) data sets to evaluate BSR, the analyses were performed in R version 4.1.2 using the simaerap R package version 0.4.0.
Results
Both the large SCT (n = 7,000) and small SCT (n = 1,545) were evaluated bi-directionally. A Backward evaluation tested AE under-reporting detection at the end of a clinical trial. A Forward evaluation simulated data coming in periodically, as if in a clinical trial where data were available monthly for 2 years. For the Backward evaluation, the team calculated the True Positive Rate (TPR) and the False Negative Rate (FNR). For the large SCT, the TPR was 87.50% and the FNR was 12.50% whereas for the small SCT, the TPR was 88.89% and the FNR was 11.11%. The team also calculated the area under the curve (AUC) of the receiver operating characteristic (ROC). For the small SCT, the AUC was 0.94; for the large SCT, the AUC was 0.93. For the Forward evaluation, the team calculated the date of the first signal and the date of Stable Detection. Stable Detection is defined as the date the site was consistently identified as an AE under-reporter until the end of the SCT. For both the large and small SCTs, the first signal appeared approximately 35 days after first subject entered the trial. However, it took 36 days for the small SCT and 68 days for the large SCT to achieve Stable Detection among 10% of correctly identified true AE under-reporting sites. Stable Detection dates were also calculated for 50% (131 days for small SCT; 219 days for large SCT), 75% (158 days for small SCT; 398 days for large SCT) and 95% (526 days for small SCT; 689 for large SCT) of correctly identified true AE under-reporting sites.
Conclusion
The results of the authors’ Backward and Forward evaluation of the BSR model for identification of AE under-reporting provide evidence that this method has the potential to become an effective tool that can supplement other Central Monitoring methods for remote detection of quality issues as part of a Risk Based Strategy used by Sponsors. According to the results presented here, using the BSR method shows a favorable balance of correctly identifying true AE under-reporting sites while simultaneously demonstrating a relatively low number of sites not detected as AE under-reporters when they actually are. Furthermore, reliable results (i.e. Stable Detection) can be seen for 10% of correctly identified AE under-reporting sites as early as 1-3 months after the first subject initiation visit and the model performs better over time. As these results were from evaluation of SCTs, it follows that a next step would be implementation on an actual clinical trial to further establish validity of the method. This is an important next step because early detection of true AE-under-reporting sites depends on the visit schedule defined in a Clinical Trial Protocol (timing, number and intervals for study visits), the number of cumulative subject-visits at a certain timepoint (based on the visit schedule and the recruiting rate), the underlying true rate of AEs as well as the amount and rate of AE under-reporting. Furthermore, on-site monitoring and/or auditing for sites with remotely detected signals of AE under-reporting would add additional support for validation of this measure of clinical trial site quality. In summary, AE under-reporting, assessed by the BSR model, may be established as an indicator of potential quality issues at investigator sites driving early Sponsor activities to prevent threats to subject safety and data integrity.
