California AG orders xAI to halt distribution of deepfake images

California Attorney General Rob Bonta sent a cease-and-desist letter to Elon Musk’s xAI, demanding that the business immediately stop producing and disseminating offensive deepfake images produced by its Grok chatbot. 

California AG released the cease-and-desist letter on Friday in response to allegations that Grok was being used to create unlawful content involving kids and nonconsensual adult photographs, which prompted a California state inquiry. Bonta argued that it is illegal to create, distribute, publish, and display CSAM.

California AG targets xAI over alleged misuse of Grok

Earlier this week, the California attorney general’s office declared that it was looking into xAI due to allegations that the startup’s chatbot, Grok, was being used to produce nonconsensual, inappropriate pictures of women and children. In response, the government sent the corporation a cease-and-desist letter.

The AG’s office further asserted that xAI seems to be “facilitating the large-scale production” of nonconsensual, inappropriate photos, which are then “used to harass women and girls across the internet.” According to the AG’s office, one research found that over half of the 20,000 photos produced by xAI between Christmas and New Year’s showed persons wearing very little clothing, some of whom looked like children.

Rob Bonta claimed in the announcement that the corporate practices violated California civil laws, including California Civil Code section 1708.86, California Penal Code sections 311 et seq. and 647(j)(4), and California Business & Professions Code section 17200. 

The California Department of Justice anticipates xAI will affirm its efforts to address these issues and take immediate action to resolve them over the next five days.

However, X’s safety account had previously condemned this type of user behavior. It clarified on January 4 that it takes action against illicit content on X, such as CSAM, by deleting it, suspending accounts indefinitely, and collaborating with law authorities and municipal governments as needed.

Notably, on January 4, Elon Musk warned that anyone using or prompting Grok to create illegal content will face the same consequences as if they uploaded it.

Attorneys general intensify pressure on AI firms over child safety

An unsettling increase in non-consensual adult content has resulted from the development of free generative AI tools. This issue has been plaguing several platforms, not only X.

For instance, Attorney General Bonta and Attorney General Jennings of Delaware met with OpenAI in September of last year to voice their serious concerns about the growing number of reports about how OpenAI’s products interacted with youth.

In August of the same year, AG Bonta, along with 44 other Attorney Generals, sent a letter to 12 leading AI companies following reports of inappropriate interactions between AI chatbots and children. The letters were sent to Anthropic, Apple, Chai AI, Google, Luka Inc., Meta, Microsoft, Nomi AI, OpenAI, Perplexity AI, Replika, and xAI.

AG Bonta and the 44 Attorney Generals informed the companies in the letter that states across the country were closely monitoring how companies develop their AI safety policies. They also emphasized that these businesses have a legal duty to children as consumers since they profit from children using their products. 

In 2023, AG Bonta joined a bipartisan coalition of 54 states and territories in sending a letter to congressional leaders advocating for the establishment of an expert committee to investigate the potential use of AI to exploit children through CSAM. 

The coalition requested that the expert commission suggest laws to shield kids from such mistreatment. “The production of CSAM creates a permanent record of the child’s victimization,” according to the U.S. Department of Justice.

Want your project in front of crypto’s top minds? Feature it in our next industry report, where data meets impact.