The user interface of the AI software involved in the case against Zheng and five others for producing obscene materials. Photo: CCTV News

Six people have been sentenced to prison in East China's Zhejiang Province for using generative AI to produce obscene content for profit after modifying an AI model to bypass its built-in safety restrictions, CCTV News reported.

The Yuyao People's Court in Zhejiang recently concluded a case where the defendant surnamed Zheng and five others were guilty of producing obscene materials for profit. The six were sentenced to prison terms ranging from three years and one month to three years and 10 months, and were also fined. The ruling has taken effect after none of the defendants appealed.

Zheng, a former programmer at a major Chinese gaming company, came up with the idea of developing a similar product after encountering an overseas AI sex-chat bot in July 2024, according to the report. He later formed a development team with a colleague and several people he met online.

In January 2025, the team launched its generative AI model on an overseas platform and charged users through several payment channels, including cryptocurrency.

Mainstream generative AI models have built-in restrictions on illegal and inappropriate content. The group used prompt engineering to "jailbreak" the base model, repeatedly rewriting system prompts, setting up character cards and creating specific dialogue scenarios to bypass the model's original restrictions on pornographic content, the court said. 

After paying, users could choose different characters to engage in sexual conversations with AI bots and spend credits to generate obscene static or animated images.

During its operation, the model generated more than 370,000 static images and over 6,000 animated images, while users paid about $24,000, or roughly 170,000 yuan, into the service, according to the court. 

Police opened an investigation in July 2025 after discovering clues related to the case and arrested all six defendants within two months.

A key issue in the case was whether developers could be considered to have "produced" obscene materials when the final content was generated only after users selected characters and entered specific instructions.

The court found that although users took part in the process, all of their actions took place within a technical framework designed in advance by the defendants. The defendants exercised substantial control over the model's output, and their conduct had a direct causal link to the production of the obscene content.

The court therefore ruled that using technical means to bypass an AI model's ethical and safety restrictions in order to generate obscene content constituted the production of obscene materials under criminal law.

A judge handling the case said that "technological neutrality" cannot be used as a shield for illegal activity, and that developers cannot avoid responsibility simply because the final content is generated by AI.

The judge also noted that AI can produce content with a single command and on a massive scale, meaning courts should not simply apply traditional quantity-based standards. Factors including the scope of distribution, risk of wider dissemination and actual harm should also be considered when determining criminal responsibility and sentencing.

Global Times