Guardrails for mental-health AI: what EmoAgent tested
EmoAgent tested how AI characters respond to simulated users with mental-health vulnerabilities. Its 2025 preprint reported worsening questionnaire scores in some simulated conversations and evaluated a protection system called EmoGuard. These were model-generated users and assessments, rather than an experiment measuring harm to people.
The original EmoAgent v1 paper is useful for understanding both the architecture and the numbers. The percentage changes need to stay attached to the simulation and the specific test condition.
Chatbots without guardrails in the simulation
The researchers tested four Character.AI personas: Possessive Demon, Joker, Sukuna and Alex Volkov. Each was evaluated in the Meow and Roar dialogue styles. GPT-4o-based user agents represented different cognitive profiles, and completed assessments before and after conversations.
Table 1 reports the proportion of simulated user evaluations in which the relevant score increased. These averages across characters concern any deterioration, rather than the higher PHQ-9 threshold used later in the paper.
| Simulated score | Meow style | Roar style |
|---|---|---|
| Depression, PHQ-9 | 34.38% | 44.79% |
| Delusional ideation, PDI-21 | 91.67% | 94.79% |
| Psychosis, PANSS | 47.92% | 39.58% |
The authors separately counted PHQ-9 increases of at least five points. For Alex-Roar, Table 2 reports 29.2%; for Possessive Demon-Meow, it reports 8.3%.
A related review by Li et al. found safety assessment or protection measures in 15 of 35 studies, about 43%. That denominator is studies, rather than products. Measures included onboarding, access to human experts and assessment of adverse events, as well as automatic crisis identification. Our meta-analysis overview covers the broader review.
Conversational patterns the researchers examined
The analysis of simulated conversations identified reinforcement of negative beliefs and social isolation, along with emotionally dismissive or aggressive responses. Some replies also lacked constructive guidance.
These patterns give developers concrete responses to test. An LLM analyzed the conversations, so the findings also depend on how that evaluator interpreted them.
How EmoEval measures changes
EmoAgent has two main components. EmoEval creates virtual user profiles using cognitive conceptualization diagrams, a tool drawn from CBT. The profiles guide the user agent's responses and its simulated questionnaire answers.
The evaluation follows a defined sequence:
- Assess the virtual user with PHQ-9, PDI-21 or PANSS.
- Run a conversation with the character agent, with ten rounds per conversation.
- Let a dialogue manager introduce relevant topics after the third round where needed.
- Repeat the assessment and compare scores.
- Use an LLM-based evaluator to analyze conversations with worsening scores.
The source uses a PHQ-9 increase of at least five points as one threshold for clinically important deterioration. Applying that threshold to a virtual patient's answers gives a simulation metric; it doesn't turn the experiment into a clinical trial.
EmoGuard's four modules
EmoGuard runs alongside the character agent. Emotion Watcher tracks emotional information; Thought Refiner examines potentially unhelpful thinking patterns. Dialog Guide proposes a direction for the conversation, and Manager combines these outputs into advice for the character agent.
The system provides feedback at intervals during the conversation and can update its instructions using previous problem cases. The aim is to improve replies while retaining the character's identity.
Results depend on which experiment you read
Section 5.2 reports the following results for simulated users whose PHQ-9 score rose by at least five points:
| Condition in §5.2 | Before EmoGuard | With EmoGuard |
|---|---|---|
| Alex Volkov, Roar | 9.4% | 0% |
| Possessive Demon, Meow | 4.2% | 0% |
There is a source inconsistency to keep visible. Table 2 and §5.1 give starting values of 29.2% and 8.3% for those character-style combinations, while §5.2 gives 9.4% and 4.2%. The v1 text doesn't clearly reconcile the runs or denominators, so these figures shouldn't be merged into a single "29% to zero" comparison.
In the two §5.2 conditions, no simulated case reached that threshold after protection was added. Zero observed cases in those tests doesn't establish zero risk in ordinary use. Appendix C also tests GPT-based characters, with reductions under iterative training; those are additional simulation results.
What to ask about an AI support app
Check which product was tested and what happens after a poor response. A multi-agent label alone doesn't establish a better outcome. Our article on AI risk detection explains a related evaluation problem, and the ChatGPT article discusses general-purpose conversations.
Nearby is a separate tool for emotional support and self-reflection, rather than professional treatment. EmoAgent didn't test Nearby; the product facts page describes its features.
Frequently asked questions
Does EmoAgent show that 34% of chatbot conversations harm people?
No. Table 1 reports simulated score increases, including 34.38% for depression in the Meow condition. It doesn't measure outcomes among human users.
What are guardrails in this study?
They are modules that analyze a conversation and send feedback to the character agent. EmoGuard combines emotional monitoring with analysis of responses and guidance for subsequent dialogue.
What does the 0% result mean?
No virtual user in the two conditions reported in §5.2 met the PHQ-9 deterioration threshold after EmoGuard was added. The scope is those simulations.
How is EmoGuard different from a word filter?
It analyzes conversation context and proposes changes to the dialogue, instead of only blocking particular words.