Five CBT chatbots compared by design and research evidence
Five CBT-related chatbots offer five useful examples of structured dialogue. Their studies answer different questions. SuDoSys tested conversation quality with simulated clients and student role-play; CRBot used real conversations reviewed by professionals; Socrates 2.0 examined four-week feasibility; a behavioral-activation chatbot used artificial users; and a GPT-4 problem-solving chatbot compared immediate user experience.
Reading them as five equivalent clinical efficacy trials would miss the useful part: which design was tested, with whom, and what the researchers actually measured.
For symptom outcomes across randomized trials, see our AI chatbot meta-analysis and comparison of rule-based and LLM chatbots. Here the unit of comparison is the individual system.
What each evaluation can tell us
CBT offers tasks that can be described step by step: examining a thought, planning an activity or working through a current problem. That makes the dialogue easier to specify and evaluate. Following those steps and improving symptoms, however, are separate outcomes.
| System | Study design and participants | Measured result | What remains untested by this design |
|---|---|---|---|
| SuDoSys, Chen et al. (2024) | 148 simulated clients; separate role-play with 20 college students | Ratings of coherence, professionalism, empathy and authenticity; a modest coherence advantage over comparison systems | Symptom improvement in people receiving the intervention |
| CRBot, Wang et al. (2025) | 19 users; conversation logs reviewed by four mental health professionals | Qualitative strengths and problems in cognitive restructuring, including leading questions and oversimplified advice | A controlled effect on symptoms |
| Socrates 2.0, Held et al. (2025) | 61 adults, four weeks, uncontrolled mixed-methods feasibility study | High feasibility and acceptability; small pre-post symptom changes; feedback on repetition and depth | Whether the tool caused the symptom changes |
| BA chatbot, Kuhlmeier et al. (2025) | Artificial users with varied profiles; dialogues assessed by clinical experts | Ratings of session structure and technique delivery; practical problems with activity planning and barriers | Benefit to real users with depression |
| GPT-4 PST chatbot, Mo et al. (2025) | 100 adults randomized to two groups of 50; 5–9 exchanges | Higher immediate ratings for problem awareness and problem solving than minimally prompted GPT-4, with no significant relationship-quality difference | Sustained symptom improvement or long-term use |
SuDoSys and CRBot make different design choices
SuDoSys uses the WHO's Problem Management Plus (PM+) guidelines to organize a conversation into stages. A stage-aware instruction generator supplies the current task; a topic database retains relevant information.
The transition mechanism is worth looking at closely. The LLM generates a status value, c, and the controller updates the stage as s := s + c. The controller executes the transition, but the model contributes the decision. This is a structured workflow, with model behavior still affecting where the conversation goes.
In the automatic evaluation, GPT-4 rated dialogues with 148 simulated clients. Separately, 20 students adopted client profiles and rated their interactions. The authors found a modest coherence advantage, with other ratings broadly competitive. The established evidence for human-delivered PM+ doesn't automatically validate this chatbot version.
CRBot focuses on cognitive restructuring: helping someone examine an automatic thought and consider alternatives. Nineteen users interacted with the chatbot, and four mental health professionals reviewed the logs.
Those reviewers saw natural conversation and useful Socratic questions. They also found leading questions, evaluative language and advice that simplified the user's situation too far. A prompt can ask an open-looking question while still steering toward a preferred answer. That is a concrete design problem the CRBot study helps identify.
Socrates 2.0 tests use over four weeks
Socrates 2.0 combines a conversational agent with supervisory agents that monitor dialogue and changes in belief. Its goal is cognitive reappraisal through Socratic questioning.
The 61 participants used it as often as they wished for four weeks, averaging 6.70 conversations. Feasibility and acceptability ratings were high. Participants reported small pre-post changes in depression, social anxiety, obsessive-compulsive and posttraumatic stress symptoms. With no control group, the study can't isolate the tool's contribution to those changes.
The interviews are useful on their own. Some participants valued the on-demand, nonjudgmental conversation. Others criticized repeated questioning and limited depth. These findings point toward improving conversational progression; they don't establish the same failure pattern as CRBot's professional review.
Activity planning and problem solving
The behavioral-activation chatbot studied by Kuhlmeier and colleagues helps users plan activities connected to their values and current capacity. In the cited 2025 preprint, artificial users supplied varied responses and clinical experts evaluated the resulting dialogues.
The assessment found useful protocol delivery alongside uneven handling of realistic plans and obstacles. For example, suggesting an activity isn't enough if the plan ignores the person's starting point or leaves practical barriers unexplored. This design tests how the chatbot responds to controlled profiles. It doesn't measure recovery in young people with depression.
Mo and colleagues' PST chatbot uses GPT-4 for problem-solving therapy: define a current problem, consider possible responses and plan a next step. One hundred adults aged 18–35 who had contracted COVID-19 were randomized to the structured chatbot or a minimally prompted GPT-4 control.
After 5–9 exchanges, they completed a user-experience questionnaire. The structured version received higher ratings for problem awareness and problem solving, with no significant difference in relationship quality. Randomization strengthens that comparison, but the outcome is still an immediate impression after a short conversation. Participants with a history of mental illness or substantial screened symptoms were excluded.
What to take from these designs
The practical lesson is to evaluate the behavior a system claims to provide. A staged chatbot needs tests of its transitions. A thought-examination tool needs review of whether its questions leave room for the user's interpretation. An activity planner needs realistic plans and follow-up on obstacles.
Those are engineering recommendations drawn from the studies, rather than a specification all five independently validated. Our articles on prompt engineering and MIND-SAFE explore ways to organize that work.
For Nearby's actual scope and features, see the product facts. Nearby offers emotional support and self-help; results from these research systems don't establish its treatment effectiveness.
Frequently asked questions
Have all five chatbots been tested for clinical effectiveness?
No. The designs range from simulated dialogue and professional review to uncontrolled feasibility and randomized user-experience comparison. Each provides a different kind of evidence, as the table above shows.
Does SuDoSys control transitions independently of the LLM?
Its controller updates the stage, but the LLM generates the transition status used in that update. The model still influences progression through the conversation.
Why do leading questions matter in a thought record?
They can push the user toward an interpretation instead of helping them examine their own evidence. CRBot's professional reviewers identified leading questions and evaluative language as concrete issues to improve.
What would establish symptom benefit?
A study of the specific system with appropriate symptom measures, follow-up and a suitable comparison group. Dialogue quality and usability help explain the experience, but they answer different questions from treatment efficacy.