P100: A Mixed Methods Study of Real World Use of a Machine Learning Application for Coding Adverse Events in Clinical Trials

Poster Presenter

Walker Bradham

Product Leader, Zelta
Merative
United States

Objectives

The first objective of this study was to identify quantitative measures for determining the effect of artificial intelligence systems on coding adverse events in clinical trials. Secondly, we summarize qualitative strengths and weaknesses of the evaluated system.

Method

System usage data from 04/09/2020-04/02/2021 were analyzed to compare number of searches required for an approved code with and without AI coding system support. Semi-structured interviews were conducted with users and evaluated for themes to assess system accuracy, efficiency, and time savings.

Results

A total of 7,965 approved codes (32% AI supported) from 14 endocrinology clinical trials (79% Phase 1, 21% closed, etc.) conducted by a single contract research organization were assessed. Codes for 4.12% of manually coded AEs were approved after a single search while codes for 72.27% of AI supported AEs were approved after 0 searches (the AI system proposes codes for AEs upon verbatim selection) and 81% after 2 searches. Manually coded AEs required more searches with 56% approved after 3 and 80% after 4 searches. Code rejection rates for each coding method were similar (1.58% manual, 2.43% AI-supported). Code rejection rates for each coding method were similar (1.58% manual, 2.43% AI-supported). Semi-structured interviews lasting 60 minutes were conducted via phone with coding system users (n=3). Users stated the AI coding system improved efficiency and decreased work effort while maintaining requisite accuracy. Additionally, the module integration and use was deemed intuitive and seamless.

Conclusion

AI systems integrated into coding workflows can increase code approval efficiency while maintaining accuracy and user satisfaction.