Avijit Ghosh only had 50 minutes before the laptop would shut down and lock him out, so he had to act fast. 

The task?

Complete as many as 21 challenges designed to get AI chatbots like ChatGPT to produce misinformation before time ran out.  

The Northeastern University graduate and lecturer was taking part in the Generative Red Team Challenge at the Defcon hacker convention in Las Vegas last month.

The goal was to find vulnerabilities in the models to help companies developing these technologies make their systems better over time. 

The competition was hosted by AI Village, a community-based nonprofit composed of hackers and data scientists, and had the backing of the Biden administration.

Ghosh knew some tricks to get the bots to “hallucinate” that he picked up reading online forums and from messing around himself. But, in reality, he didn’t know what to expect. 

“The [prompt] questions weren’t shared with me beforehand, so it was as new to me as anybody else,” he says. He ended up placing eighth on the leaderboard, he says, and he is overall happy with his performance, getting the models to produce both political misinformation and decisions based on prejudiced class systems.   

Ghosh, who graduated from Northeastern with a doctorate in computer science in June and is teaching a course on ethical machine learning this fall, did more than just compete in the challenge. 

He was also a volunteer and will co-author a report on what flaws, inaccuracies, weaknesses and vulnerabilities were found during the 2½-day competition, which drew more than 2,200 competitors from all over the world. 

Among the attendees were 220 students from 18 states, according to AI Village. 

The report won’t be out for a while though, and Ghosh can’t disclose specific information. It is under embargo for six months to give the companies that are developing these technologies the opportunity to fix any security issues, Ghosh says. 

“We are going through millions of data points, and in six months, if not longer, we will publish a report that details what vulnerabilities we found,” he says. “I personally hope to look into the ethical concerns, but there will also be security experts, machine learning experts and theoretical researchers.”

The goal of the report is to serve as an “easy-to-access resource for everybody to see what problems exist and how we can combat them,” he said in an interview with The New York Times. 

Ghosh is quick to recognize though that a report alone is not enough. 

“You have to have accountability that has teeth,” he says, noting that legislation is one of the best ways to make technology companies more accountable. 

He noted that it was a great sign that the Biden administration supported Defcon, and highlighted that the event is helping both politicians and the general public understand how these technologies work. 

Ghosh has been fascinated with the intersection between ethics and technology for much of his life. 

For his doctorate thesis at Northeastern, he focused on algorithmic fairness and the challenges and considerations that come with building machine learning models that are equitable and just.  

This semester, he is teaching a course on building responsible machine learning models at the Boston campus.  

“I hope that out of that class, my students who go on to work in tech companies, or whatever they do, that whenever they build systems, they will be more considerate about what has happened in the past,” Ghosh says.
Out in industry, he is working as a research data scientist at AdeptID, a Boston-based AI company that uses machine learning to help candidates find jobs.

“There’s a tension between people who are architects that want to build these systems and people who want these systems to be ethical and fair,” Ghosh says. “But I don’t think these are incompatible. In my research, I found that accuracy and fairness don’t always have to be compromised for each. If you make a system fair, it doesn’t mean it will be less accurate.”

At his job at AdeptID, which he’s had since July, Ghosh is in charge of making sure the AI models the company builds are fair, unbiased and meeting certain policies and government regulations.

Ghosh regularly speaks on the topic. He recently was invited to speak with the Centre for Data, Ethics and Innovation, a division of the United Kingdom’s Department of Science, Innovation and Technology, about one of his recent papers. 

He encourages Boston policymakers to speak with members of the AI community like himself more regularly. 

“I want to work with policymakers to shape policy,” he says. “I wish Boston city officials reached out to people like myself to work in AI policymaking.” 

The way Ghosh sees it, government regulations are needed to help keep companies developing AI systems in check. 

At the same time, he says, it’s important to continue to invite a diverse set of voices to help improve these technologies over time.

“We need both,” Ghosh explains. “We need a community approach and a top-down regulation approach to control the harms of AI.”

Cesareo Contreras is a Northeastern Global News reporter. Email him at c.contreras@northeastern.edu. Follow him on Twitter @cesareo_r.