P246: Automated Extraction of Chemistry, Manufacturing, and Controls (CMC) Change Information from Unstructured Reports
Poster Presenter
Ray Kirchmier
Manager
Deloitte United States
Objectives
Develop an automated system for identifying and analyzing CMC Changes from unstructured documents. The solution must include a centralized platform for accessing and evaluating CMC data, enabling more efficient analysis processes and enhanced decision-making capabilities.
Method
The multi-step approach:
• Artificial intelligence capabilities parsed unstructured text into machine-readable format
• Conducted keyword identification and pattern recognition data analysis to identify relevant context
• A Large Language Model (LLM) extracted CMC changes from relevant context
Results
From the large corpus of documents on record, this LLM solution extracted CMC changes from over 417k+ relevant pages (from 50k+ documents) with a 95% recall rate and 84% precision. The automated system improves efficiencies with detecting relevant contents and provides the first ever centralized repository of CMC changes within an analytics dashboard. At its core, this dashboard provides direct access to summarized CMC change information in a structured format, including immediate navigation capabilities to the respective source materials that CMC contents were extracted from. The CMC change summaries generated by the LLM are listed in an easily digestible format and can be assessed by users at multiple levels (e.g., document, page). Users can filter CMC changes by multiple criteria and can search through extracted CMC change summaries for keywords or phrases, to return isolated changes data pertaining to specific areas of analysis interest.
Conclusion
These reports contain multiple documents, comprise hundreds to thousands of pages, and are reported in an infinite combination of unstructured text and formats (e.g., lists, tables, paragraphs). Previously, it was a highly manual process to navigate robust sets of information across these reports to 1) locate the relevant CMC change information and 2) interpret the CMC changes and understand their implications. With the implementation of this solution, many users have streamlined access to CMC change information that is more comprehensible, significantly saving hours spent annually. In addition, this fully automated solution is refreshed with new report data daily, enabling timely access to new insights, thus mitigating review bottlenecks and backlogs. Since these CMC changes are now available and can be analyzed in one location, this solution also enhances the organization’s ability to assess the implications of certain CMC changes, which may assist in subsequent activities or other decision-making processes.
