DSM5AgentFlow and AI diagnosis on a synthetic benchmark
DSM5AgentFlow tests whether language models can connect a simulated interview to DSM-5 criteria and show the basis for their prediction. In Ozgun and colleagues' study, all 8,000 therapist-client conversations were AI-generated. The results describe performance on that synthetic benchmark, rather than diagnostic accuracy with real patients.
Qwen-QWQ achieved reported overall accuracy of 70% and F1 of 77%. Some individual class F1 scores exceeded 93%. Those measures answer different questions, so combining them into a headline promising "94% accuracy" would change the meaning of the result.
How the three agents work
The therapist agent turns the DSM-5 Level-1 Cross-Cutting Symptom Measure into conversational questions. The measure covers 23 questions across 13 symptom domains. The client agent responds according to a predefined profile, with symptoms and demographic details assigned for the experiment.
The diagnostician agent then processes the transcript. It retrieves relevant passages from DSM-5, predicts a disorder label and generates a report that includes its rationale and treatment suggestions. The dataset spans 10 primary disorder categories, with 2,000 conversations generated for each of four tested language models.
This design makes it possible to compare models against known profile labels. It also means the models generate both sides of the conversation. The authors produced conversations in a single pass, which doesn't test genuine turn-by-turn adaptation to a person's changing answers.
The multi-agent design separates responsibilities. Whether that improves a particular deployed system requires a comparison of that system; assigning agent roles isn't itself evidence of better diagnosis or therapy.
RAG makes evidence available for inspection
Retrieval-augmented generation, or RAG, supplies the diagnostician with the five most relevant DSM-5 passages from a document index. The model combines them with the transcript when preparing its answer.
The output can mark symptoms with <sym> tags, quote dialogue with <quote> and identify medical information with <med>. These markers help a reader inspect which evidence the answer uses. A retrieved passage can still be misapplied, and a well-formatted explanation can still contain an incorrect inference.
That difference is visible in the paper's explainability assessment. The authors inspected one randomly selected transcript per model and counted evidence tags, criterion references and reasoning steps. In the Qwen-QWQ example, they found 11 symptom tags and four direct quotes, with DSM criteria and a numbered rationale. GPT-4.1-Nano produced more tags but no DSM clause references or step list in its selected example.
This is a small case study of explanation structure. It doesn't establish that every answer from one model is transparent or correct.
Read the metrics separately
The tested models were Llama-4-Scout-17B, Mistral-Saba-24B, Qwen-QWQ-32B and GPT-4.1-Nano. Section 5.2 reports these overall results:
| Model | Reported measure | Result |
|---|---|---|
| Qwen-QWQ | Overall accuracy | 70% |
| Qwen-QWQ | Overall F1 | 77% |
| GPT-4.1-Nano | Precision | 83% |
| GPT-4.1-Nano | Overall F1 | 73% |
Precision describes how many predicted positives are correct; recall describes how many relevant cases the model finds. F1 combines precision and recall. Accuracy measures correct classifications across the evaluated set. A precision of 83% therefore can't be relabeled as 83% accuracy.
Table 3 then reports F1 for each disorder category. Qwen-QWQ scored 93.65% for panic, 94.36% for PTSD and 93.89% for social anxiety. GPT-4.1-Nano's PTSD F1 was higher, at 98.53%. These are category-specific benchmark scores, rather than an overall probability that a diagnosis is right.
Adjustment disorder shows why the distinction matters. F1 ranged from 0.98% for GPT-4.1-Nano to 40.25% for Qwen-QWQ; the other two models scored 2.78% and 2.75%. The confusion matrices showed frequent mix-ups with depression. Strong performance on one label can coexist with very poor performance on another.
What the study contributes
The useful contribution is a reproducible workflow for linking a predicted label to interview text and retrieved criteria. The code and data let other researchers inspect that process and test alternatives. Clinical usefulness would require evaluation with real interviews and outcomes.
A diagnostic label also answers a different question from a model of a mental process. Our computational psychiatry overview explains approaches that study learning and perception. Karvelis, Paulus and Diaconescu (2023) discuss the reliability challenges for those computational measures; that's a separate measurement problem, not a test of DSM5AgentFlow.
For another type of observation, see digital phenotyping. Neither article supplies evidence that DSM5AgentFlow validates a commercial app. Nearby's purpose is emotional support and self-help, described in the product facts, rather than DSM-based diagnosis.
Frequently asked questions
Did DSM5AgentFlow reach 94% diagnostic accuracy?
The values around 94% are F1 scores for selected categories in a synthetic benchmark. Qwen-QWQ's reported overall accuracy was 70%, with overall F1 of 77%.
What does DSM-5 add to the workflow?
It provides diagnostic criteria for retrieval and a symptom questionnaire for constructing the simulated interview. The model must still connect the text and criteria correctly.
How thoroughly were explanations evaluated?
The authors examined one randomly chosen transcript per model. They counted tags, quotations, references to criteria and reasoning steps. That's an illustration of output structure, with a small sample.
Which category was hardest in this benchmark?
Adjustment disorder had F1 scores from 0.98% to 40.25% across the four models and was frequently confused with depression. The result helps identify a weakness to investigate in further testing.