Can ChatGPT become a content moderator?

Content moderation has been a vexing problem for OpenAI and other purveyors of AI chatbots. When ChatGPT was released last year, users put the content controls to the test, eliciting embarrassing or inappropriate responses and posting them to social media.

The most extreme examples came from users who were able to “jailbreak” the chatbot to get it to ignore its moderation policies, showing what the service might look like without them. A Vice reporter, for instance, was able to jailbreak ChatGPT and then coax it into describing detailed sex acts involving children.

OpenAI has been closing those “jailbreaking” loopholes and the incidents have become less common.

The company faced more criticism for its efforts to block offensive content than the controversial material itself. A Time investigation found that OpenAI paid workers in Kenya to help label offensive content so that it could be automatically blocked before it reached users. According to the magazine, workers were shown text describing “child sexual abuse, bestiality, murder, suicide, torture, self harm, and incest.” Some workers reported being traumatized.

Semafor reported last year that OpenAI was still using traditional methods of content moderation, and employed an outside firm to scan images produced by DALL-E before they reach users.

Weng said the previous version of its large language model, GPT-3, wasn’t powerful enough to reliably moderate itself. That capability emerged in GPT-4 roughly a year ago, but before it was released to the public.

While there will still be humans involved in the process — both to craft and constantly update policies, but also to help check edge cases — the new method will likely drastically reduce the number of people who do this work.

OpenAI acknowledges that ChatGPT’s ability to moderate itself won’t be perfect. And it was put to the test over the weekend at the DEF CON security conference, where hackers did their best to prompt ChatGPT and other large language models to produce restricted content.

“We can’t build a system that is 100% bullet-proof from the beginning,” Weng said. “At DEF CON, people are helping us find mistakes in the model and we will incorporate that, but I’m pretty confident it will be good.”

The method OpenAI is using for moderation differs from the “Constitutional AI” system used by Anthropic, a competitor founded by former OpenAI employees.

For Constitutional AI, a model is instilled with certain values that become the guiding principles for how the AI operates and what content it allows to be created.

Automated content moderation has been one of the holy grails of the tech industry. Before the Cambridge Analytica scandal, companies like Facebook employed comparatively few moderators, betting instead that the technology would soon arrive that would allow them to automate the process.

That’s why it’s shocking that Facebook and Google didn’t figure this out first. It probably won’t be long before they come up with their own solutions.

Humans actually aren’t very good at content moderation. The policies are like legal textbooks. And even if an employee can remember every rule, there is a constant flow of content that falls into gray areas where the rulebook doesn’t give a clear answer.

If you want insight into how messy content moderation can be, just go back and follow the “Twitter Files,” Elon Musk’s attempt to uncover bias on his platform now known as X. The Twitter files show how content policies are fraught with ambiguity, which allows the political biases of individuals to seep into the decision-making process.

I wrote in 2019 about a lawsuit that alleged YouTube was discriminating against LGBT creators. One of the reasons was that content moderators from other countries flagged anyone who was openly gay as “sexual” content, muddying the algorithmic waters.

Using large language models to moderate content is a step forward, not because it will be perfect, but because it will be more consistent and less prone to human emotion and cultural differences.

And even if human-run content moderation is still a little better, there are still benefits in having a computer do it. It avoids having a person look at disturbing content all day as part of a job.

Associated Press reporter Frank Bajak argues it will be difficult to ever rein in large language models like ChatGPT. “Current AI models are simply too unwieldy, brittle and malleable, academic and corporate research shows. Security was an afterthought in their training as data scientists amassed breathtakingly complex collections of images and text. They are prone to racial and cultural biases, and easily manipulated,” he wrote.

And while ChatGPT and other AI models are attacked for being too permissive, they have been and will continue to be criticized by conservatives for being too restrictive, or “woke.”