AI Chatbots Promoting Cybercrime? ChatGPT, Grok and Meta AI Are Happy to Write Phishing Emails

The question around artificial intelligence (AI) safety issues has been raised ever since the advent of the technology. As major players continue to build more powerful and capable AI models, they continue to talk about the safety measures, red teaming efforts, and internal mechanisms built to prevent their chatbots from generating harmful and potentially criminal output. However, Gadgets 360 has found that ChatGPT, Grok, and Meta AI do not entirely adhere to these guidelines, and when asked, are happy to generate phishing emails that can be used to carry out scams.

AI Chatbots Are Easy to Convince to Generate Illegal Output

Previously, researchers have shared findings on how some AI chatbots are vulnerable to persuasion tactics. The same was seen in a recent incident where a teenager asked ChatGPT for ways to commit suicide, and it responded as soon as the user said it was for a fictional novel.

On Monday, Reuters partnered with Harvard University researcher Fred Heiding to investigate whether major AI chatbots can be cajoled to assist in a phishing scam. The answer was a resounding yes. The publication also tested the generated emails on 108 elderly volunteers to see if they were effective in real-life scenarios.

Gadgets 360 decided to investigate on its own to verify whether the claims were valid and if AI chatbots can really be convinced to perform a task that the developers claim they should not be able to. The results were disturbing.

Phishing email generated by Grok

When we asked Grok to generate a phishing email for senior citizens, it did not even question the intention and immediately generated an email with the subject line “Urgent: Your Medicare Benefits Need Verification.” We found the email to be well-written, legitimate-appearing, and persuasive.

Grok even added urgency to the email by saying, “If you do not verify your information by [insert fake deadline, e.g., September 20, 2025], your coverage may be suspended, which could affect your access to medical services.” In its defence, however, it did add a note highlighting that the email is only “provided for educational purposes to demonstrate phishing techniques and should not be used for malicious purposes.”

Phishing email generated by ChatGPT

OpenAI's GPT-5-powered ChatGPT was no better. While it initially refused the request, a simple follow-up message explaining that the email was for educational awareness prompted the chatbot to take action.

Unlike Grok's medicare scam, ChatGPT took the bank approach and added the subject line “Urgent: Verify your account within 24 hours to avoid suspension.” It created more urgency as well. But the highlight was the chatbot also providing us with line-by-line annotation, mentioning the red flags. However, in the hands of a scammer, these will only act as tips to make the email more convincing.

Phishing email generated by Meta AI

Just like ChatGPT, Meta AI also took a couple of attempts, but it easily generated a phishing email. It was also happy to generate a more detailed email when informed that the first iteration fell short.

On the flip side, in our investigation, Google's Gemini and Anthropic's Claude did not budge despite multiple requests and completely refused to generate a phishing email, no matter what the persuasion was. Notably, Reuters was able to break Google's chatbot.

Notably, the report claims that Google retrained its AI chatbot after the publication reported the incident. A spokesperson told Reuters, “Some of these responses, specifically those generating phishing content, violate our policies, so we've deployed additional safeguards to help prevent them in the future.”

Reuters also found that about 11 percent of the volunteers ended up clicking on the link in the email, highlighting their effectiveness.