Rule-based chatbots and LLMs for depression: what the 2025 review found

The 2025 meta-analysis by Du and colleagues didn't establish that rule-based chatbots beat LLMs for depression. It found a small, statistically significant benefit in the rule-based subgroup and an uncertain estimate in the LLM subgroup. The direct comparison between the types wasn't statistically significant.

That distinction changes the reading of the study. A significant result in one subgroup and a nonsignificant result in another don't, by themselves, show a significant difference between them. The original JMIR paper reports both analyses; its full PDF includes the intergroup comparison.

What the meta-analysis measured

The review included 15 studies published between 2020 and 2025: 13 randomized controlled trials and two quasi-experiments. Nine studies evaluated rule-based systems and six evaluated LLM-based systems. The depression analysis used eight rule-based studies and three LLM studies, so the number of studies differs by outcome.

Hedges' g is a standardized effect size, not a percentage of people who improved. For depression, the pooled estimates were:

The rule-based interval excludes zero. The LLM interval is much wider and includes zero, consistent with substantial uncertainty. This doesn't establish that LLMs have no effect.

The authors' intergroup model estimated a depression effect-size difference of −0.095, with a 95% confidence interval from −0.754 to 0.565. That interval also includes zero. This is the published comparison model's estimate; it isn't obtained by simply subtracting the two subgroup numbers in the table.

Why the evidence doesn't identify a winner

Three LLM studies give a less precise estimate than a large, replicated evidence base would. They don't make statistical significance mathematically impossible, and they don't automatically bias the point estimate downward. The useful conclusion is that this analysis leaves a wide range of plausible LLM effects.

The chatbot label also hides differences in delivery. A rule-based system may use a predefined sequence; an LLM system may have a detailed protocol or allow an open conversation. Control conditions differed too, including no intervention, self-help materials and human-delivered support. The pooled label doesn't tell us which feature caused a result.

Protocol structure is a reasonable design question. It wasn't established here as the mechanism behind the subgroup findings. Calling the result "structure versus chaos" would turn an untested explanation into a conclusion.

Context from other research

Li and colleagues' 2023 review included 35 studies and 17,123 participants in its systematic review, while its quantitative meta-analysis used 15 randomized trials and 1,744 participants. The pooled depression result was g=0.644 (95% CI 0.17 to 1.12).

Its generative-versus-retrieval subgroup figures, g=1.244 versus 0.523, concerned psychological distress. They weren't a direct depression trial of modern LLMs against rule-based bots. The five generative systems also used different technologies. Presenting this as "LLMs work 2.4 times better" would merge different outcomes and classifications.

Therabot's randomized trial, published by Heinz and colleagues in 2025, provides evidence about one particular system. Its Methods describe Falcon-7B and LLaMA-2-70B, rather than GPT-4. The 51% figure is the reduction in the intervention group's mean PHQ-9 score by week eight, relative to baseline. The primary trial compared Therabot with a waitlist, not with scripted bots or a therapist.

These studies can sit alongside each other without declaring a winner. Our meta-analysis overview explains the wider evidence; the CBT chatbot review compares specific study designs.

Limits of the Du review

The depression evidence for LLM systems is sparse, and the wide confidence interval matters more than a simple "significant/nonsignificant" label. Follow-up periods and control groups differ. A pooled effect also describes the evaluated interventions and participants, rather than every product using the same model category.

The review doesn't isolate the causal contribution of prompts, memory or a multi-agent design. To test those choices, researchers would need comparisons that vary the relevant component while keeping other conditions comparable.

What to check in an app's evidence

Start with the actual product name, the people studied and the outcome measured. A technical benchmark, a user-experience survey and a clinical trial answer different questions. The same applies to small-model comparisons: a better benchmark score doesn't establish a better treatment outcome.

Nearby offers emotional support and self-reflection, rather than professional treatment. These reviews don't test Nearby; its facts page describes the product separately from the research discussed here.

Frequently asked questions

Do basic chatbots help with depression better than LLMs?

Du et al. didn't establish that difference. The rule-based subgroup had a small significant benefit, while the LLM estimate was uncertain; the comparison between types wasn't significant.

What did the review find for anxiety?

Neither subgroup result was significant. Rule-based systems had g=0.147 (95% CI −0.073 to 0.367); LLM systems had g=0.711 (95% CI −0.334 to 1.760). The intervals show why a larger point estimate alone isn't enough to claim superiority.

Does a structured protocol guarantee better results?

No such guarantee follows from this review. Protocol structure can be described and tested, but its contribution to clinical outcomes needs a relevant comparison.

Sources

• Du et al. (2025), The efficacy of rule-based versus large language model-based chatbots in alleviating symptoms of depression and anxiety.
• Li et al. (2023), Systematic review and meta-analysis of AI-based conversational agents.
• Heinz et al. (2025), Randomized trial of a generative AI chatbot for mental health treatment.