For people struggling to overcome addiction, good therapy can be hugely beneficial. But bad therapy is worse than no therapy at all.

The challenge is being able to tell the difference. Practicing counselors rarely receive feedback, so it’s difficult for them to know if they are providing optimal treatment to their patients.

Northeastern professor Tad Hirsch is working to change that. He has developed a system that records counseling sessions and rates them, generating a report card for therapists.

The technology is already in use at a university training clinic and a network of opioid addiction treatment facilities. Once scaled up, the system could create a new standard of care for therapy and lead to improved mental health for patients all over the world.

Here’s how it works: Hirsch and his collaborators started by recording 356 counseling sessions. From those sessions, they pulled out 300,000 statements and compiled them into a database. Then a team of psychology experts combed through the statements and coded them.

Clinicians described the technology as accurate, insightful, and useful. They also thought the tool was particularly valuable for training, as a way of providing feedback to counselors as they were getting certified.

Tad Hirsch, Northeastern professor

Coding is an analytical process in which data are categorized. In this case, the experts coded statements based on how closely they aligned with techniques used in motivational interviewing, a type of psychotherapy.

Next, researchers used the coded dataset to train a machine learning model. This is a form of artificial intelligence. Once the model is trained, the result is a system that can record a counseling session, create a transcript of what was said, analyze the transcript, grade the session, and provide specific feedback to clinicians.

In other words, a computer is evaluating therapists and telling them how they can improve.

How do therapists react to being assessed by a machine? Hirsch and his collaborators recently completed a study to find out, testing the technology with 21 counselors. The results were overwhelmingly positive.

“We found that across the board, they all saw value in what we were doing,” said Hirsch, a professor of design at Northeastern whose research focuses on the intersection of design, engineering, and social justice.

“Clinicians described the technology as accurate, insightful, and useful. They also thought the tool was particularly valuable for training, as a way of providing feedback to counselors as they were getting certified,” Hirsch said.

Therapists also expressed a high degree of trust in the machine learning model. Hirsch said this is likely because advances in artificial intelligence have been covered extensively in the news, and the public generally has a positive view of the technology. Plus, as Hirsch said one therapist put it, “It’s hard to argue with a computer.”

And while the machine learning model evaluates therapy sessions with about 90 percent accuracy compared to a human expert, Hirsch cautions that the system is not infallible. And the stakes are high.

“This is different than when Netflix makes a movie recommendation that you don’t happen to like,” Hirsch said. He encourages therapists to speak up if they disagree with the machine’s assessment and to use the report card as a springboard for honing their practice.  

“As designers, we want to ensure that the predictions our models are making are contextualized, such that people can understand how these systems are working well enough to interpret findings and results they might be seeing, rather than just take them at face value,” Hirsch said.