Why AI companies are turning their chatbots over to hackers

The scale and transparency of this exercise, and the participation of so many creators of large language models like ChatGPT, is notable. And it makes sense why the companies would want to play ball: They aren’t paying to participate in this weekend’s challenge, organizer Rumman Chowdhury said, so they’re essentially getting a mass volume of testing and research for free. Plus, the White House is keeping an eye on it.

What matters more is what happens after this weekend. The companies, as well as independent researchers, will receive the results of the competition as a massive database, which will detail the various issues found in the models. It’s ultimately on them to fix the problems, and a report due to come out next February will include whether they did so.

“I wouldn’t necessarily take it on faith” that the companies will fix every problem that emerges, said Chowdhury, an AI ethics and auditing expert. “But we are creating an environment where it is a smart idea to be doing something about these harms.”

The skillset of a large language model “red teamer” is completely different from that of the traditional hacker set, which focuses on bugs and errors in code that can be exploited. A coding mindset can be helpful in figuring out how to trick these AI models into slipping up, but the best exploits are done through natural language.

“We’re trying something very wild and audacious, and we’re hopeful it works out,” Chowdhury said.

One thing the hackers won’t be testing for: partisan bias. While chatbots became a part of the culture wars this year, with some conservatives claiming they’re “woke,” Chowdhury said that’s largely the result of trust and safety mechanisms, not the models themselves.

“We’re not really wading into that water,” she said. “These models are not fundamentally politically anything.”

One of the big questions for large language models is whether the harmful content can be “watermarked” so that social media companies can easily identify and stamp it out. Right now, that looks like a huge challenge in text and a slightly less daunting one in AI-generated images and video.

There’s an argument that it’s not all that important what large language models do when they’re being used by individuals. What matters more is whether the content can be distributed on a mass scale through social media and other channels.

People can already generate plenty of offensive and inaccurate content. LLMs help people do that better in a more convincing tone and at scale.

And people have demonstrated en masse how this technology can be manipulated, acting as a kind of public red team army by calling out chatbots that go awry. The most prominent example was when a New York Times reporter coaxed Bing Chat into falling in love with him. Also, the companies already do their own internal red-teaming and acknowledge their chatbots can produce offensive or incorrect information.

In Europe, the biggest concern about large language models is that they might violate copyright laws. The models were created by ingesting massive amounts of text and transcribed audio. And the technology could be used to displace some of the people who created that underlying content, without competition.

It would be interesting to see if the red teamers at DEF CON or future chatbot hackers could get LLMs to generate content that appears to violate copyrights or borrow content from specific content creators.