“It totally blew my mind.” That’s what graduate student Laura Pfeifer Vardoulakis said of her encounter with work taking place in Timothy Bickmore’s lab in the College of Computer and Information Science. Bickmore is one of the few researchers starting to develop medical technologies that target patients and individuals instead of clinicians. “When I came here I was still thinking about it from the doctor’s side,” said Vardoulakis, who worked at a medical software company before coming to graduate school. “He was working on it from the patient side and it blew my mind.” Bickmore’s team designs computer-based “relational agents,” or animated computer characters that look and act like humans. They’re nurses and other health care workers that live inside the computer screen and do everything from walking patients through hospital discharge papers to providing companionship to older adults. They stand to do a lot for preventative medicine but Vardoulakis realized there was still more to be done to optimize their impact.

“The two main issues are how do you keep people using the technology–or wanting to use it–over a long period of time, and then, two, how do you get good quality data if you’re not using a sensor?” A lot of things, such as questions about emotional state, fall into this category of stuff that can’t be easily quantified. Vardoulakis wanted to design a study that could shed light on whether relational agents could help in some of these more abstract issues. Perhaps a series of questions presented by an avatar would be more compelling than a simple, internet-based text survey. If so, then perhaps avatars would be a good tool for longitudinal studies in which participants are asked to answer questions at regular points over a long time period. But she also wanted to look at some other factors as well. For instance, if survey participants are offered money are they more likely to keep coming back and answering the questions each week? Or what if they’re offered meaningful feedback about their responses, would that do the trick?

After she conducted several studies with older adults in the Boston area, for her thesis Vardoulakis decided to conduct a larger-scale study with the community right outside her lab door: college students. The number one health concern for college students is alcohol consumption, so she chose to make this the topic of her survey. Starting with a standard questionnaire used by the Office of Prevention and Education at Northeastern, or OPEN, Vardoulakis designed an online survey as well as a relational agent that asked the participants questions directly with its automated voice. Half of her 375 participants used the agent, while the other half used the text survey. She also randomly assigned each person to one of two feedback groups and one of two monetization groups. Each time they returned for another week of inquisition, half of the participants would receive a completely non-judgemental analysis of their survey answers the previous week, while the other half received no feedback All of the participants were entered into a drawing for a $25 amazon gift card each week, but only half were reminded of the drawing each time they returned.

So, what did all of this tell Vardoulakis? First, she found that two things seemed to impact a person’s adherence to the study. First, the duration of their first session was the biggest determinant. Regardless of the delivery method and whether or not they got informational feedback or reminders about the gift card drawing, participants who took less than 200 seconds to answer the first survey were more likely to stick with it over the duration of the 16-week study.

The second determinant was whether or not the participant was a drinker. If they reported drinking at least one alcoholic beverage during the course of the study, Vardoulakis checked them off as drinkers. In the text group, drinkers were more likely to stick with it than non-drinkers and vice versa for the agent group. When it comes to self reported data, it’s well known that people tend to do something called “maximizing their social desirability.” “We’re more likely to up-play our positives and downplay our negatives,” said Vardoulakis. This makes some amount of sense when interacting with a real human…we don’t like to be judged. But the same thing seemed to be happening with the relational agent. There should have been no statistical difference between the text and agent group, but the fact that there was suggests that participants may have been feeling subconsciously judged by the computer screen character. I can understand that — Bickmore’s team has spent years trying to make their agents as life-like as possible. They even study real clinicians and try to emulate the facial expressions and mannerisms people use when interacting with patients.

Vardoulakis wanted to believe that feedback would be a powerful tool to promote adherence as well as providing quality data. Unfortunately, she said, it kind of turned out the opposite. Participants who were told they would receive feedback about their answers tended to report fewer alcoholic beverages than the non-feedback group. They were also less likely to report negative consequences as a result of their drinking (e.g., missing a class, getting into a fight, etc). Keep in mind this was on their very first encounter with the survey, so they haven’t even seen that feedback yet, they just know it’s gonna happen some day. “It’s almost a social desirability issue, where you’re presenting yourself to yourself,” said Vardoulakis.

Overall, the study provided some good information for people designing novel patient-facing healthcare technologies. It’s obviously very important for people to maintain anonymity when dealing with sensitive subjects like alcohol use, this has been known for a long time in the field. But developers may not have realized just how important. Social desirability isn’t only relevant in the face of our fellow humans, it’s also apparently a factor when we’re dealing with machines.

The field of patient-facing healthcare technologies is positioned to make a big impact on the field. But in order for any of these programs and devices to be truly valuable, especially from a healthcare perspective, they need to be supported by reliable studies. I think that this one from Vardoulakis drives that point home. We might have some strong biases about what we expect to be successful but until we put it out there and test it, we can’t say a thing. Of course, Vardoulakis’ work is only a beginning and addresses a very specific population. More work needs to be done in the same vein to show exactly where and how these tools can be useful.

And what about those first two questions she asked herself (how do we keep people coming back and how do we ensure they provide meaningful data)? Well,  a handful of things: For one, first impressions matter – the assessments should not take a lot of effort to complete, said Vardoulakis. Secondly, when dealing with potentially sensitive health topics, text-based interfaces are preferable over agent-interfaces.. Third, providing feedback might backfire – people might sensor themselves more if they know they’ll receive feedback. And finally, although I didn’t mention it above, the monetary incentive was powerful in getting people to complete the survey once, but its effect wore off rapidly and didn’t seem to matter long-term.