P243: A Generative AI Tool for Searching and Generating Responses to Regulatory Authority Inquiries on Clinical Trial Notifications
Poster Presenter
Takayuki Yasuda
Statistical Programmer
Chugai Pharmaceutical Co., Ltd. Japan
Objectives
To improve the efficiency of the inquiry process for Clinical Trial Notifications (CTN) submitted at the start of clinical trials in Japan, we constructed and evaluated a chat-based tool for searching past CTN inquiry responses and generating responses using retrieval-augmented generation (RAG).
Method
We built a retrieval-augmented generation (RAG) system for searching inquiry responses held internally on an AWS cloud environment and evaluated the accuracy and usefulness of its search results and response generation via questionnaires administered to test members.
Results
We conducted two rounds of evaluations of the developed RAG system with 12 members involved in the CTN process. Using inquiry items and response files from approximately 50 past trials as the search dataset, we collected participants' opinions on the accuracy, business applicability, and usability of the tool in retrieving past responses and generating new draft responses for inquiries from three designated trials. In the first evaluation, we met the predefined criteria for (1) ensuring that the responses did not contain incorrect information (criterion: over 70% rated the responses as useful; first round result: 77% rated them as useful) and (2) reducing work hours (criterion: fewer than 50% of respondents indicated that the system did not reduce work hours; first round result: 33% reported no reduction). However, the criteria were not met for (3) the system’s usability as a response proposal to regulatory authorities (criterion: over 70% endorsement; first round result: 58% rated it as usable) and (4) respondents' desire for its implementation (criterion: over 70% endorsement; first round result: 58% expressed a desire for implementation). After the first evaluation, we improved the system prompt settings and refined the RAG output method, and subsequently conducted a second evaluation. The second evaluation results were: (1) 86% rated the responses as useful, (2) 14% reported no reduction in work hours, (3) 86% rated the system as usable, and (4) 86% expressed a desire for implementation. All predefined criteria were met, suggesting the system’s overall usefulness.
Conclusion
In this initiative, we developed and evaluated a system employing retrieval-augmented generation (RAG) for searching past responses and generating draft responses to improve the efficiency of the CTN inquiry process in Japan. Although the initial evaluation did not fully meet all predefined criteria, subsequent system customization improved its business applicability and overall usefulness, as demonstrated by the second evaluation. The system has already been integrated into the company's interactive generative AI tool and is currently used in actual CTN processes. In future work, we plan to further enhance its accuracy by implementing the latest large language models and optimizing prompts, and we aim to expand the application of this approach to inquiry response processes for other regulatory documents.
