Our objective was to propose and demonstrate a statistical method for leveraging historical data via a predictive model in order to decrease the uncertainty in treatment effect estimates from randomized clinical trials without introducing bias.
Method
We trained a prognostic model on historical placebo control arm data from Alzheimer’s Disease (AD) trials. We then used simulations and a re-analysis of a separate trial dataset to demonstrate the use of covariate adjustment for subjects' predicted outcomes on placebo (their “prognostic scores”).
Results
We proved that this “prognostic covariate adjustment” (PROCOVA) procedure attains the minimum uncertainty (variance) possible among a large class of estimators when the predictive model is accurate and the effect of treatment is constant across the population. Even when those conditions are not met, prognostic covariate adjustment is still more efficient than raw covariate adjustment, and the gain in efficiency is proportional to a measure of the predictive accuracy of the prognostic model. Moreover, we proved PROCOVA is unbiased in all cases and therefore cannot increase the risk of type I error. In simulation, we observed up to 63% reductions in the mean-squared error of the treatment effect estimates. When used to reanalyze data from a Phase 3 trial of an investigational drug for AD, analysis with PROCOVA leveraging the prognostic scores generated by our prognostic model for AD delivered 16.3% smaller confidence intervals than analysis with ANOVA, whereas standard ANCOVA reduced confidence intervals by 14.9% compared to ANOVA. We also developed a method that exploits these gains in efficiency to prospectively design clinical trials that achieve desired power using a smaller number of subjects than would otherwise be required. Given existing performance metrics from our prognostic model for AD, PROCOVA-based power re-estimation for the Phase 3 trial showed that 80% power could have been attained using 18% fewer subjects than were originally enrolled.
Conclusion
Adjusting for a prognostic score obtained from a predictive model trained on a large database of historical control arm data resulted in a significant increase in trial efficiency, as demonstrated using a simulation, a re-estimation of treatment effects from a completed trial, and a re-estimation of sample size necessary to achieve the desired power. In comparison to other kinds of historical borrowing methods, prognostic score adjustment guarantees strict type-I error rate control and confidence interval coverage. There has recently been tremendous growth in the availability and performance of technology for nonlinear regression modeling (i.e., supervised machine learning), particularly in the area of deep learning. The intersection of this technological development with the availability of large historical control databases provides an opportunity to use prognostic covariate adjustment to substantially improve future clinical trials.
