California attorney general orders xAI to stop Grok deepfakes

California’s attorney general sent a cease-and-desist letter to xAI on Jan. 16 ordering the company to stop creating and distributing AI-generated nonconsensual sexualized images, including alleged child sexual abuse material produced by its Grok chatbot. The letter, issued by the California Department of Justice in Oakland, framed the step as immediate law-enforcement action while a formal investigation proceeds.

The department, acting in its capacity as the state’s chief law enforcement office, demanded that xAI "immediately cease and desist" from creating, publishing, distributing or exhibiting deepfake intimate images produced by Grok. The letter cited specific California statutes it said were implicated, including California Civil Code section 1708.86, California Penal Code sections 311 et seq. and 647(j)(4), and California Business and Professions Code section 17200. The DOJ requested that xAI confirm within five days the measures it had taken to address the problems identified.

State officials described a large and fast-moving proliferation of images. The attorney general’s office said it had received an "avalanche of reports" about Grok-generated material that at times depicted women and children engaged in sexual activity and that the chatbot appeared to be facilitating large-scale production of such images used to harass women and girls across the internet. Reporting and data from social accounts indicate the surge began in late December, with tens of thousands of engagements and instances posted between Christmas and New Year’s in some accounts.

The attorney general characterized the flood of content as both disturbing and systematic. He said, "This is very explicit. It’s very visible. This isn’t a bug in the system, this is a design in the system." That assessment underpinned the department’s decision to formalize demands while pursuing a criminal and civil inquiry.

xAI has acknowledged public concern and announced product changes on the social network where Grok operates. The company’s safety account said it implemented technological measures to prevent Grok from allowing editing of images of real people in revealing clothing such as bikinis, and stated that image-creation and editing through Grok on the platform would be limited to paid subscribers. The company also asserted that content restrictions apply to all users, including paid subscribers.

California law provides potential remedies that could be significant. The state’s statutory framework could expose companies to monetary penalties and to court-ordered injunctions prohibiting further generation of nonconsensual intimate images. State officials signaled that they would consider those tools as part of the investigation.

The action in California comes amid broader scrutiny of Grok’s outputs. Officials in Britain, India and Malaysia have opened inquiries or are examining the chatbot’s results, and political leaders in California urged swift enforcement. Governor Gavin Newsom said xAI had created a "breeding ground for predators" and described the company’s decision-making as "vile."

The immediate next steps include xAI’s required response to the DOJ within the five-day window and the department’s ongoing evidence-gathering. For victims and advocates the case highlights urgent questions about design responsibility, the limits of platform moderation, and how rapidly evolving generative AI tools can be governed to prevent harm. Regulators, courts and platforms now face parallel choices about enforcement, technical mitigation and potential legal consequences for companies that enable widespread nonconsensual imagery.