Clinical Communication Simulation

Role
Led research; collaborated on product direction
Team
3-person student team, with a national medical education board's Lead NLP Scientist
Timeline
12 weeks
Tools
ChatGPT, Lovable
Role
Lead researcher
Team
3-person team with an NLP scientist
Timeline
12 weeks
Tools
ChatGPT, Lovable

TL;DR

Problem
Medical students rarely get to practice hard patient conversations, because human actors are costly and the Step 2 CS exam ended in 2021.
Approach
Studied how clinical communication is taught, then tested an early prototype with 5 medical students and practitioners.
Solution
An AI patient for on-demand practice, paired with a separate AI evaluator for feedback.

01.Examining the gap in communication practice for medical students

Standardized patients (actors trained to portray real patients) are the gold standard for practicing clinical communication, but they're costly and hard to scale.

Better patient outcomes are linked to provider interpersonal skills, yet medical students get few chances to practice difficult conversations and little feedback, because human actors are limited. The USMLE Step 2 Clinical Skills exam, a major standardized test of future doctors' patient communication skills, was also discontinued in 2021. Could AI help close that gap?

How might we use AI to help medical students practice interpersonal communication and receive constructive feedback, on demand and at scale?

02.Defining realistic patient behavior & testing an early prototype

To create a supportive AI tool, we studied existing evaluations and student needs, then built a quick custom GPT to explore an early interaction model. We brought this V1 prototype to 5 medical students and practitioners, then worked with our stakeholder on the system design.

Together we designed against two LLM tendencies that threatened assessment validity: (1) sycophancy, where the model agrees with the user and inflates their scores, and (2) role drift, where the patient slips into acting as the evaluator.

Insights from V1

What students said
  • “It seems hard to actually address and reassure, which is frustrating since the patient feels the same throughout the encounter - feels more like a clinical test with a challenge rather than a challenge to overcome for motivational interviewing.”
  • “…it would feel more realistic if the VSP [virtual standardized patient] provided us basic labwork, that way our conversation with the patient is more meaningful/reflects a real scenario.”
Random OSCE patient case, simple medical complexity.
Knocks lightly and sits down, shifting a bit in the chair.
“Hi... um, thanks for seeing me. I’ve been having this sore throat and stuffy nose for about three days now, and it’s just not getting better. My throat feels scratchy, and swallowing kind of hurts. I’ve also been feeling really tired, like I just want to sleep all day. I don’t think I’ve had a fever, but I’ve felt kind of warm at night. I’m mostly worried because I have an important presentation at work tomorrow, and my voice sounds awful. clears throat softly I’m not sure if this is just a cold or something more serious, and I don’t really know when I should be concerned.”
I understand why that would worry you, thank you for coming in. How long have you been feeling tired for?
Thinks for a moment, rubbing my eyes a bit.
“Probably about... four days now? I noticed it around the same time my throat started bothering me. At first I thought I was just staying up too late, but even when I sleep a full night, I still wake up feeling drained. By the afternoon I’m pretty wiped out, which isn’t normal for me. I usually have decent energy, even with work. It’s frustrating because I feel like I’m moving in slow motion.”
I'm sorry to hear that you haven't been feeling like yourself. Let's take your temperature

03.Building realistic patients & a trustworthy evaluator

Testing V1 with students, practitioners and our stakeholder showed what the platform needed to be both useful and effective. Both sets of needs became our design requirements.

Students and practitioners needed realistic virtual patients to practice across different communication skills, and feedback that aligned with their learning goals. Therefore, the platform must provide defined patient personas that challenge different communication competencies, give patient details up front instead of a vitals interview, and end each conversation with a detailed report on what to improve.

Our stakeholder needed a system that could scale and be trusted as an educational tool. Therefore, the system must have architecture that supports both patient and evaluator without drifting roles, with the evaluator limited to approved frameworks.

Design responses

  1. user interaction

    Scenario selection

    The student picks a scenario to practice with.

  2. Choose a patient

    Students pick the patient they want to practice with. Each is designed to challenge different communication competencies.

  3. system

    Configuration layer

    The selected scenario is sent to the patient agent as its system prompt, which generates the patient persona in real time and the door note. A specific prompt for each scenario keeps the agent in its patient role and guards against role drift.

The Training Scenarios screen: six patient cases, each tagged with the competency it trains, Foundational Communication marked Recommended First
      The practice interface mid-conversation: the transcript between the student and the AI patient, the patient overview beside it, and the message input with voice and send controls under it
      1. user interaction

        Practice conversation

        The student talks with the patient, who pushes back when empathy cues are missed.

      2. Door note

        Patient details appear before the encounter, matching the clinical workflow and replacing the vitals interview.

      3. system

        Patient agent

        The patient stays in character, so it does not simply agree with the student (sycophancy) or drift out of the patient role.

      1. user interaction

        Feedback report

        The student reads the evaluator's feedback on the conversation.

      2. Evaluation report

        After the student finishes the conversation, a report renders. Each category has a written rationale and areas to improve.

      3. system

        Evaluator agent

        A separate evaluator, limited to the transcript, scores against approved frameworks, to keep feedback valid and the model from drifting between patient and evaluator roles.

      The OSCE Evaluation Report open over the chat screen: an overall performance score, then a detailed assessment by category, each with its score, a written rationale and areas to improve

        04.Looking ahead

        Mock video calls. Text-only chat can't show a patient's expressions or body language. Future scope would include an avatar for a mock video call, so students can practice reading those cues the way they would in a real visit.

        Validate the evaluator agent. The AI evaluator's scores haven't been checked for validity. We would compare the AI evaluations against an expert human evaluator to measure model performance and feedback validity.