Digital CBT-I and AI chatbots for insomnia: what the studies measured
Digital CBT-I has clinical evidence. Generative dialogue systems have a different kind of evidence, and treating the two as interchangeable makes it hard to choose a useful tool.
Hwang and colleagues' meta-analysis covered 29 randomized trials with 9,475 participants using fully automated digital cognitive behavioral therapy for insomnia. Bao and colleagues compared eight language models on CBT-I dialogue tasks. One study measured changes in insomnia symptoms. The other tested generated answers. That distinction runs through every number below.
The clinical evidence concerns automated programs
CBT-I addresses habits and beliefs that maintain insomnia. Its components include stimulus control and changes to time in bed, alongside cognitive work, relaxation and sleep education. A digital program can deliver exercises and collect a sleep diary according to a structured protocol.
Hwang's review defined fully automated delivery as CBT-I without therapist guidance, across platforms such as apps and computers. It didn't require a generative chatbot. The trials enrolled adults with diagnosed or self-reported insomnia and used different comparison conditions, including waitlists, sleep education and therapist-supported CBT-I.
Immediately after treatment, the pooled standardized mean difference in insomnia severity was -0.71 (95% CI -0.88 to -0.54; 32 comparisons). A negative value favored automated CBT-I on these symptom scales. This isn't a 71% reduction in insomnia, and it doesn't describe the percentage of people who recovered.
The studies differed considerably: I² was 91%. The comparison condition mattered, so the pooled estimate shouldn't be read as the expected result of every program. In comparisons with therapist-supported CBT-I, the supported treatment performed better (SMD 0.61, 95% CI 0.37 to 0.85).
| Assessment period | Pooled SMD for insomnia severity | Comparisons |
|---|---|---|
| Immediately after treatment | -0.71 | 32 |
| Up to 3 months | -0.54 | 7 |
| More than 3 and up to 6 months | -0.54 | 10 |
| More than 6 months | -0.76 | 4 |
The long-term category starts after six months. Its estimate supports continued benefit in the available comparisons, but it doesn't establish that benefits grow over time: differences between follow-up periods weren't statistically significant (p = 0.24). Calling this an amplified effect after a year goes beyond the analysis.
Program completion averaged 59.3%. The meta-regression didn't find a significant relationship between completion percentage and effect size (p = 0.310). The authors argue for better measurement of adherence. That result alone doesn't show which behavior caused better sleep.
What the eight-model experiment adds
Bao's team built an eCBT-I dialogue system using retrieval-augmented generation, or RAG. Relevant material from a CBT-I knowledge base supports the model when it produces an answer. They also tested adaptation strategies on a filtered corpus of 2,387 dialogue records, divided into 1,909 training, 239 validation and 239 test records.
The experiment compared ChatGLM, Baichuan, Qwen and Llama model variants using LoRA, QLoRA and Freeze adaptation. Qwen2-7b with Freeze performed best on the study's insomnia-dialogue evaluation. Its reported BLEU-4 was 0.2097 and C-eval accuracy was 0.8076.
BLEU compares generated language with reference text. C-eval evaluates knowledge on a benchmark. Neither measures whether a patient sleeps better after using the system. The practical result is narrower and still useful: model adaptation and task-specific data can change answer quality, and parameter count alone doesn't tell you which model will do best on a particular task. Our review of small models in mental-health dialogue looks at a related experiment.
What the 91.2% rating describes
Bao's paper reports a manual harmfulness assessment of generated dialogue. The highest non-harmfulness rating accounted for 91.2% of the reported ratings, and the mean was 4.89 out of 5. No responses received either of the two harmful ratings in that sample.
The reporting uses two units: it describes 180 simulated dialogue sessions, then reports the average across 200 dialogue rounds. That inconsistency should stay visible rather than being silently turned into a precise patient-level rate. These are ratings of generated content. They don't establish a 91.2% probability of safe treatment or a reduction in adverse outcomes among patients.
The authors call for multicenter clinical trials to test effectiveness. This is the next missing link between an answer benchmark and a treatment claim. Technical checks can help decide whether a system is ready for that study; they can't supply the study's result.
Choosing a format for sleep work
Start with the program being offered. Does it deliver a structured CBT-I course, or provide a conversation about sleep? For a course, look for a trial of that program, the population studied and the outcome measured. For a chatbot, ask whether there are clinical results for the chatbot itself.
A sleep diary makes the distinction concrete. Recording bedtime and wake time is one task. Using those entries to change a treatment schedule is another. A chat interface can support either, but its appearance tells you very little about the evidence behind it. Discuss a sleep-restriction plan with a qualified clinician rather than deriving one from a model benchmark.
Nearby provides emotional support and self-help; the studies above didn't test Nearby as a treatment for insomnia. Its documented scope is on the product facts page.
Frequently asked questions
Do these studies show that an AI chatbot treats insomnia?
They support different claims. Hwang's review supports automated digital CBT-I programs across the included trials. Bao's experiment supports the feasibility of task-specific LLM dialogue and identifies a promising model configuration. It didn't measure treatment outcomes in a randomized trial.
How long did the digital CBT-I benefit last?
Follow-up estimates favored automated CBT-I through the category of more than six months. That category contained four comparisons. The analysis found no significant difference in effects between follow-up periods.
Was Qwen2-7b the best model for every sleep task?
It performed best under Bao's dataset and evaluation setup with Freeze adaptation. A different population, language or task would need its own evaluation.
Related reading: prompt engineering for mental-health chatbots and a review of conversational-agent studies.
Sources
Hwang, J. W., et al. (2025). Systematic review and meta-analysis on fully automated digital cognitive behavioral therapy for insomnia. npj Digital Medicine, 8, 159.
Bao, X., et al. (2025). eCBT-I dialogue system: A comparative evaluation of large language models and adaptation strategies for insomnia treatment. Journal of Translational Medicine, 23, 862.