At a timepoint of 4 weeks into a 9-week treatment with the use of a relatively simple classification algorithm, treatment failure in Internet-Delivered Cognitive Behavior Therapy for Insomnia (ICBT-i) can be predicted with a balanced accuracy of 67%, according study findings published in Internet Interventions.1
A key question with the use of an Adaptive Treatment Strategy is “how accurate the prediction needs to be before one can act upon it.” In Adaptive Treatment Strategies, each patient’s outcome is predicted early in the course of treatment, with treatment adapted to those at risk for failure. The researchers of the current clinical trial (ClinicalTrials.gov Identifier: NCT01663844) sought to establish a minimal empirically supported level of accuracy for a classifier to be clinically useful in an Adaptive Treatment Strategy. They also explored and compared prediction models with varying complexity and implementation potential.
In order to attain these goals, the following 3 specific aims were cited by the researchers: (1) to determine the accuracy of the randomized controlled trial (RCT) classifier used to predict treatment failure in a prior proof-of-concept study, thus creating a benchmark for empirically supported minimal accuracy in future developments of Adaptive Treatment Strategies; (2) to examine the relative value of each of the predictors for the classifier; and (3) to explore the additive value of different logical sets of predictors.
Data for the current study were derived from a prior published RCT,2 in which patients who underwent ICBT-i were classified as “Red” (risk of failed treatment) or “Green” (not at risk) during treatment week 4 of 9 weeks. In the current study, the term “Green” is used to denote those patients who were predicted to experience a good outcome (treatment success) based on the algorithm, and the term “Red” is used to denote those patients who were predicted to experience a bad outcome (treatment failure), according to the algorithm.
The classification took place at week 4 for 2 reasons: (1) week 4 was the point at which patients who were working at the prescribed pace will have gone through all of the psychoeducation and rationale, and will have initiated sleep restriction; and (2) the researchers still wanted as much time left in treatment as possible to adapt the treatment and assist those individuals who they thought would not benefit sufficiently with their current trajectory.
Overall, half of the “Red” patients were randomly assigned to receive an adapted treatment. The current study evaluated data from all of the “Green” patients (n=149) and from only those “Red” patients who did not receive adapted treatment (n=51), thus enabling assessment of the accuracy of the classifier without the treatment adaption as a confounder. The study was conducted at the Internet Psychiatry Clinic — a psychiatric specialist care clinic within public health care in Sweden.
Study findings revealed that the final classification from the proof-of-concept study, which proved to be of clinical value, had a balanced accuracy of 67% and a lower bound confidence interval that was well above 50% (chance). Overall, 11 of the 21 predictors correlated significantly with Failure. A model that used all of the predictors was able to explain 56% of the outcome variance, with simpler models explaining between 16% and 47% of the outcome variance.
Key predictors included patient-rated stress, change in depression, treatment credibility, and symptoms of insomnia at week 3, as well as clinician-rated attitudes toward homework and sleep medication.
A key limitation of the current study is the fact that the sample size was selected to detect clinically relevant group differences in an RCT, rather than to evaluate classification performance. Additionally, the researchers did not examine all of the available data, and instead, examined only the data that were used by the classification algorithm.
The researchers concluded that “Simpler predictive models showed some promise and should be developed further, possibly using machine learning methods.”
Disclosure: None of the study authors has declared affiliations with biotech, pharmaceutical, and/or device companies.
1. Forsell E, Jernelöv S, Blom K, Kaldo V. Clinically sufficient classification accuracy and key predictors of treatment failure in a randomized controlled trial of Internet-delivered Cognitive Behavior Therapy for Insomnia. Internet Interv. Published online June 25, 2022. doi: 10.1016/j.invent.2022.100554
2. Forsell E, Jernelöv S, Blom K, et al. Proof of concept for an adaptive treatment strategy to prevent failures in internet-delivered CBT: a single-blind randomized clinical trial with insomnia patients. Am J Psychiatr. Published online January 30, 2019. doi:10.1176/appi.ajp.2018.18060699