status: past project
institution: university of floridaThe goal of my work on virtual humans and message production was to investigate whether virtual human interviews can elicit changes in a communication skills learners’ message production. To evaluate message production and selection interfaces in virtual human interviews, I worked with instructors of healthcare courses to integrate virtual patient interviews into their classrooms. Students interviewed multiple virtual patients, and their message production was compared over time. One aim of this project was to improve patient adherence by training doctors’ communication skills. A patient’s adherence can be affected by a doctor’s communication skills if a doctor does not communicate in a way that is comprehensible to the patient (i.e. minimizing complex language and the use of medical terminology).
I worked on this project for my dissertation research in the Virtual Experiences Research Group at the University of Florida.
Skills Applied
- Literature reviews to identify cognitive load theory
- Coordinating with stakeholders
- Survey design and qualitative analysis
- Quantitative analysis of user log data (including natural language processing techniques)
Problem
The major problem with VPF2’s current VP interviewing interface was a lack of guidance. The system did little to provide hints or constraints on what students should ask a VP. This problem can be summed up in the follow research question:
- How can a VP interview interface be designed to instruct students on the medical interviewing process while also constraining the user input to reduce frustration?
After conducting some initial research into theories of instructional design, I found cognitive load theory, which suggested that learners can benefit from studying partially solved problems (i.e. the completion effect). Using the completion effect as inspiration, I determined that one potential solution to VPF2’s lack of guidance could be to provide students with a selection interface that provided a list of questions the students could choose from. While allowing students to select questions from a list would give them examples of questions while also limiting their input, the selection process would be a technical step back in terms of realism: no doctor chooses questions from a list to interact with a patient! So, my secondary research question addresses less realistic selection interface:
- Where do healthcare students think a selection interface would fit in their learning journey, if at all?
Stakeholders
Primary users: healthcare students who are new to interviewing patients and need practice in communication skills
Stakeholders: instructors of healthcare courses that include communication skills learning
Stakeholders: instructors of healthcare courses that include communication skills learning
Roles and Responsibilities
I was the sole researcher and developer for this project. Based on feedback from healthcare education collaborators, I designed the new Guided Selection interface for VPF2, developed it in C# and Javascript, and conducted a user study with twenty-four remote users.
Scope
The main constraint of this project was the integration of the user study into a real healthcare course. To compare students’ interviewing using the regular chat interface and my new Guided Selection interface, I collected student interview data over the course of two years to get enough participants in each condition I planned to study. I also had to keep the study tasks, apart from the VP interviews, fairly short to be mindful of students’ time around assignment deadlines and exams. Based on the time limit, I opted for a single twelve-question survey after students’ final VP interview to elicit their feedback on my interface.
Process
To identify how students should be guided during a VP interview, I consulted with a subject matter expert (SME) who had experience with both medical education and VPs. This SME worked as a Nurse Educator at Shadowhealth, a company that develops VPs for use in universities and colleges. She reported that students often had difficulty in structuring their interviews and recommended a number of existing frameworks (PQRST, OLDCARTS, etc.) with which I could structure the new VP interface.
Using these frameworks, I designed a new interface for VP interviewing in an existing application, VPF2. In this new interface, instead of allowing students to type whatever they would like to ask the VP a question, students were instead asked to select questions from a menu. This design incorporated the interviewing frameworks by only showing students a select number of questions at a time. The ordering of these questions aligned with the topic ordering proposed by the interviewing frameworks.
Using these frameworks, I designed a new interface for VP interviewing in an existing application, VPF2. In this new interface, instead of allowing students to type whatever they would like to ask the VP a question, students were instead asked to select questions from a menu. This design incorporated the interviewing frameworks by only showing students a select number of questions at a time. The ordering of these questions aligned with the topic ordering proposed by the interviewing frameworks.
I then worked with a real healthcare instructor in speech-language pathology to integrate a number of VP interviews into her course. In 2018, I collected data of students using the regular VPF2 chat interface for three different VP interviews. Then, in 2019, I collected the same data from the new class using the same VPs. However, the second interview for the 2019 class was conducted using my Guided Selection interface. I compared the 2018 and 2019 students’ interviews by writing custom Python scripts to pull relevant metrics I had identified based on research in message production, a communication theory that describes the process by which a speaker forms what he/she will say.
Results
- Virtual human interviews seem to elicit changes in communication skills learners’ message production.
- Selection interfaces do not seem to affect advanced communication skills learners’ message production.
- Novice and advanced communication skills learners view selection interfaces as appropriate for novice learners.
