Menu Back to Poster-Presentations-Details

P125: State Space Models Outperform Moving Averages for Assessing Patient-Reported Quality of Life Over Time

Poster Presenter

Benjamin Ogorek

Chief Data Scientist
Spencer Health Solutions
United States

Objectives

For intermittently administered pulse surveys with multi-item scales, we wish to compare linear state space models to simple moving averages in both utility and predictive accuracy.

Method

Patients from US and Canada (348 total) used a smart hub to provide daily survey responses. Patients were randomly assigned to a train or test set, and a dynamic factor model (in state space form) was fit to the responses. Predictive accuracy was compared to 15-, 30-, and 60-day moving averages.

Results

For each of the 348 patients, months to years of daily survey data was collected, and responses to three emotional health questions were coded -1, 0, or 1, based on the sentiment of the ternary response. Optimization of the state space model’s parameters (state variance, measurement variances, and factor weights) was successful using a holdout set and a 1-ahead squared error loss function. The state space model was used to provide factor scores for emotion (via the Kalman Filter), which were available for every day regardless of the gap in responses. For the moving averages (MAs), a forward filling strategy was used for survey gaps that exceeded the window length. When comparing the 1-ahead predictions from the state space model with those from the 15-, 30- and 60-day moving averages (where simple linear regression was used to map the moving averages to responses), the state space model’s mean squared errors (MSE) were smaller in every case (p < .0001). Using unweighted averages of the three questions, the state space model’s reduction in MSE was 27% vs the 15-day MA, 19% vs the 30-day MA, and 17% vs the 60-day MA.

Conclusion

Linear state space models are convenient to use thanks to the tractability of the Kalman Filter, but their performance could be questioned when applied to a dynamic factor model for discrete data. Using holdout data sets to choose parameters, the state space model outperformed three moving averages of considerably different length. Mean squared error reduction was smallest for the moving average with the longest window, but the improvement from increasing the length from 30 to 60 days was much less than from 15 to 30 days, indicating diminishing returns and suggesting there does not exist a performance crossover point. Whereas moving averages must be modified to make predictions about future responses, the state space model implemented by the Kalman Filter readily provides predictions. Finally, the state-space model behaved well and is unambiguously defined for long periods of no response, whereas the moving averages must be artificially extended. Linear state space models are a powerful tool for analyzing sequential survey responses, and despite violations in distributional assumptions, outperform naïve estimators such as the moving average.