Menu Back to Poster-Presentations-Details

P202: Analysis Model Development with Patient Data in Atopic Dermatitis using Automated Machine Learning

Poster Presenter

Tomohiko Takahashi

Associate Manager, Headquarters of Clinical Development
Otsuka Pharmaceutical Co., Ltd.
Japan

Objectives

The objective was to develop an analysis model leveraging automated machine learning (ML) with patient data of clinical trials of difamilast for atopic dermatitis (AD).

Method

We developed an integrated dataset with two phase 2 and two phase 3 clinical trials, and then developed analysis models with automated ML. The permutation importance was utilized to select the important patient characteristics, and automated ML developed analysis models with selected features.

Results

Difamilast, a selective phosphodiesterases-4 inhibitor, was approved in Japan as a novel ointment drug to treat the patients with AD. In Japan, two placebo-controlled phase 2 trials and two placebo-controlled phase 3 trials were conducted to obtain regulatory approval. Based on the preliminary analysis results, a sub-population of pediatric patients (less than 7 years old) was excluded from the analysis dataset. The dataset consisted of patient demographics and baseline examination data including the Investigator’s global assessment (IGA), the Eczema Area (EASI), the patient-oriented eczema measure (POEM), and AD body range in 741 patients ranging from 7 to 70 years old. Since the primary endpoint of the clinical trials was reduction of IGA with two points or more after 4 weeks treatment, it was assigned as the target for analysis model. The dataset was enhanced by including five dummy parameters. The permutation importance of a certain parameter was evaluated by comparing the difference in the evaluation metric score between the original and reordered data. The median value of permutation importance for each patient characteristics, as calculated by the ten selected machine learning models, were used to identify which features were selected for further model development. With features selected, automated ML generated numerous models and compared with the models with AUC values. The best analysis model developed was an elastic-net blender model consisting of Naive Bayes combiner classifier and other six algorithms combined by automated ML. The AUC, F1 score, sensitivity, and precision of the most optimized model was 0.8100, 0.6255, 0.7488, and 0.5371, respectively.

Conclusion

In this project, the analysis model development with patient data and automated ML was assessed with actual clinical trial data. The dataset used in this project was consisted of only 741 patient data which was lower than the number of data generally required for analysis model development with ML. It is well known that having a small amount of data may lead to unstable model development by ML. To avoid this problem, the number of patient characteristics was reduced from 50 to 15 by using a permutation importance and then placing reduced characteristics into an analysis model development with automated ML. Another problem in model development with a small amount of data is overfitting. However, as this project was intended to develop an analysis model for existing data, overfitting was not relevant. Automated ML developed the analysis models and compared the analysis models with AUC values. We had over three models with 0.8 of higher AUC values, even though the amount of data for analysis model development was quite limited. All four clinical trials clearly show the efficacy of difamilast, which was considered a positive influence on the analysis model development. These results suggested that the implementation of permutation importance to reduce patient characteristic data prior to analysis model development was an important process given the smaller amount of data. The method developed in this project could be a potential tool to identify important features contributing to the efficacy in clinical trials, and could ultimately lead to enhancing the probability of success of clinical trials. In Japan, there are numerous clinical trials conducted with a small sample of patients, hence this type of ML approach could be useful when planning for Japanese clinical development.