Days after the Indian authorities sought technical particulars from Elon Musk-owned X associated to the misuse of Grok, the social media platform has warned customers in opposition to prompting the built-in AI chatbot to generate unlawful content material.
“Anybody utilizing or prompting Grok to make unlawful content material will undergo the identical penalties as in the event that they add unlawful content material,” the official account of X’s security staff stated in a submit on Sunday, January 4. Musk reiterated this in his reply to a submit saying that Grok can’t be held chargeable for person prompts.
In the course of the previous week, a number of customers on X have been commenting on photos of individuals – normally girls – by tagging Grok and asking the chatbot to “put her in a bikini” or “take her gown off”.
Story continues under this advert
The AI chatbot, which auto-replies when tagged, complied with these person requests and generated non-consensual sexual photos of each celebrities and non-celebrities, together with some who seemed to be younger youngsters.
“We take motion in opposition to unlawful content material on X, together with Little one Sexual Abuse Materials (CSAM), by eradicating it, completely suspending accounts, and dealing with native governments and regulation enforcement as essential,” the platform’s security staff stated. xAI, Musk’s AI startup that has developed Grok, additionally prohibits the sexualisation of kids in its acceptable-use coverage.
We take motion in opposition to unlawful content material on X, together with Little one Sexual Abuse Materials (CSAM), by eradicating it, completely suspending accounts, and dealing with native governments and regulation enforcement as essential.
Anybody utilizing or prompting Grok to make unlawful content material will undergo the… https://t.co/93kiIBTCYO
— Security (@Security) January 4, 2026
That is removed from the primary time that Grok has change into embroiled in an issue. The ‘anti-woke’ AI chatbot’s ‘unhinged’ mode sparked chaos in India final yr, when it replied to customers’ posts with expletives and misogynistic slurs in Hindi slang. It has additionally beforehand praised Adolf Hitler, re-shared conspiracy theories, and made inappropriate feedback in response to customers’ posts on X.
Story continues under this advert
Nevertheless, this seems to be the primary occasion of Grok’s image-generation capabilities being misused at scale. The incident extends past Grok, and underscores how generative AI continues to decrease the obstacles to create non-consensual sexually express imagery and youngster sexual abuse materials (CSAM).
It additionally raises deeper, unresolved questions on who ought to be held chargeable for dangerous or unlawful AI-generated content material, whether or not AI firms like xAI are publishers or intermediaries, and the way regulators plan to rein within the scores of shadowy ‘nudify’ platforms that might be more durable to police than mainstream AI chatbots and instruments.
Govt discover to X
Paying attention to Grok being abused to generate objectionable footage and movies of girls, with out their consent, the IT Ministry stated it has despatched a directive to X which accused the social media platform of not adhering to the nation’s legal guidelines and raised purple flags over X’s “critical failure” to implement safeguards.
Story continues under this advert
“Grok AI developed by you and built-in and made out there on the X platform, is being misused by customers to create pretend accounts to host, generate, publish or share obscene photos or movies of girls in a derogatory or vulgar method with a view to indecently denigrate them,” the IT Ministry stated within the discover despatched to the social media firm’s chief compliance officer on Friday, January 2.
The IT Ministry directed X to submit an in depth motion taken report inside 72 hours, protecting particular technical and organisational measures adopted by X in relation to Grok AI, the function and oversight exercised by its India chief compliance officer, and motion taken by the corporate in opposition to offending content material, customers and accounts.
It additionally directed the corporate to undertake a complete technical, procedural and governance-level assessment of Grok AI, together with its prompt-processing, output technology, and picture dealing with and security guardrails, “to make sure that the appliance doesn’t generate, promote or facilitate content material which incorporates nudity, sexualisation, sexually express or in any other case illegal content material”.
Story continues under this advert
The IT Ministry’s intervention got here days after Shiv Sena (UBT) MP Priyanka Chaturvedi stated that she had penned a letter to Union Minister for Electronics and Info Know-how Ashwini Vaishnaw on the misuse of Grok to generate such photos of girls.
“There must be guardrails put in place by options like Grok in order that they don’t violate girls’s dignity. Large tech corporations must take the onus. It’s not simply restricted to sharing images by pretend accounts however are additionally focusing on girls who submit their very own images. That is unacceptable and a gross misuse of an AI operate,” Chaturvedi wrote in her letter.