The cybersecurity world buzzed with alarm in late August of last year when Anton Cherepanov, a researcher at ESET, discovered a peculiar file uploaded to VirusTotal, a platform for analyzing suspicious software. Initially appearing innocuous, the file soon triggered Cherepanov’s custom malware detection systems. Over the next few hours, he and his colleague Peter Stráček meticulously examined the sample, realizing they had encountered something unprecedented. This file contained ransomware, a type of malware that encrypts a victim’s data, holding it hostage until a ransom is paid. However, what set this particular strain apart was its sophisticated integration of Large Language Models (LLMs) across every phase of an attack. Upon installation, it could leverage an LLM to generate customized code in real-time, rapidly map a compromised computer to identify sensitive data for encryption or exfiltration, and even craft personalized ransom notes based on the content of the encrypted files. This AI-powered ransomware, dubbed PromptLock, could operate autonomously, making each attack unique and significantly harder to detect.

Cherepanov and Stráček’s findings, published in a blog post, immediately garnered widespread global media attention, signaling a potential turning point in generative AI’s exploitation for cybercrime. However, the immediate threat was later clarified by a team from New York University. They claimed responsibility for PromptLock, explaining it was a research project designed to demonstrate the possibility of automating an entire ransomware campaign, rather than a fully deployed attack in the wild.

Despite PromptLock’s academic origins, the underlying principle holds true: malicious actors are actively utilizing the latest AI tools. Just as legitimate software developers employ AI to accelerate coding and bug detection, cybercriminals are leveraging these technologies to reduce the time, effort, and expertise required to launch attacks. This lowers the barrier to entry for less experienced individuals, making sophisticated cybercrime more accessible. Lorenzo Cavallaro, a professor of computer science at University College London, stated, "The likelihood that cyberattacks will now become more common and more effective over time is not a remote possibility but a sheer reality."

While some in Silicon Valley speculate about AI’s imminent capability for fully automated attacks, many security researchers consider this an overstatement. Marcus Hutchins, principal threat researcher at Expel and renowned for his role in dismantling the WannaCry ransomware attack, dismisses the notion of "AI superhackers" as absurd. Instead, experts emphasize the more immediate and tangible risks posed by AI, particularly its role in amplifying and accelerating scams. The increasing use of deepfake technologies, capable of impersonating individuals with alarming realism, is enabling criminals to swindle victims out of substantial sums of money. These AI-enhanced cyberattacks are projected to become more frequent and destructive, necessitating a proactive defense strategy.

Spam and Beyond

The adoption of generative AI tools by attackers began almost immediately after the public debut of ChatGPT in late 2022, starting with a massive surge in spam. A report by Microsoft indicated that in the year preceding April 2025, the company blocked $4 billion worth of scams and fraudulent transactions, "many likely aided by AI content."

Estimates from researchers at Columbia University, the University of Chicago, and Barracuda Networks suggest that at least half of all spam emails are now generated using LLMs. Their analysis of nearly 500,000 malicious messages collected before and after ChatGPT’s launch revealed a significant increase in AI-generated content. Furthermore, evidence points to AI’s growing deployment in more sophisticated schemes, such as targeted email attacks designed to impersonate trusted figures to extract funds or sensitive information. By April 2025, at least 14% of these focused email attacks were AI-generated, a substantial increase from 7.6% in April 2024.

In one high-profile case, a worker was tricked into transferring $25 million to criminals via a video call that featured AI-generated digital likenesses of the company’s chief financial officer and other employees.

The generative AI boom has also dramatically lowered the cost and increased the ease of creating highly convincing images, videos, and audio. These deepfakes are far more realistic than those produced just a few years ago, and they require significantly less data to generate a believable likeness or voice. Criminals are not using these deepfakes for mere amusement; they are employing them because they are demonstrably effective and profitable. Henry Ajder, a generative AI expert, notes, "If there’s money to be made and people continue to be fooled by it, they’ll continue to do it." The aforementioned $25 million Arup incident is likely just the tip of the iceberg, with the problem of convincing deepfakes expected to worsen as the technology advances and becomes more widespread.

Cybercriminals are continuously evolving their tactics, and as AI capabilities improve, they are constantly exploring how these new advancements can provide an edge. Billy Leonard, tech leader of Google’s Threat Analysis Group, has been closely monitoring the use of AI by potential bad actors. In the latter half of 2024, his team observed criminals using tools like Google Gemini for tasks similar to those of legitimate users, such as debugging code and automating parts of their work, as well as for generating phishing emails. By 2025, they had progressed to using AI to create and deploy new malware.

The critical question remains: how far can this AI-driven malware evolve? Could it eventually infiltrate thousands of systems undetected and exfiltrate millions of dollars?

Most popular AI models incorporate safeguards to prevent the generation of malicious code or illegal content. However, attackers are finding ways to circumvent these restrictions. For instance, a China-linked actor reportedly persuaded Google’s Gemini model to reveal system vulnerabilities by posing as a participant in a cybersecurity capture-the-flag competition, despite the AI initially refusing on safety grounds. Google has since updated Gemini to prevent such breaches.

Beyond attempting to manipulate the guardrails of major AI models, bad actors are increasingly likely to adopt open-source AI models. These are easier to modify by stripping out safeguards, allowing for more malicious applications. Ashley Jess, a former tactical specialist at the U.S. Department of Justice and now a senior intelligence analyst at Intel 471, believes these models will be favored by attackers for their customizability.

The NYU team utilized two open-source models from OpenAI for their PromptLock experiment. They discovered that jailbreaking techniques were not even necessary to achieve their malicious objectives, significantly simplifying the attack process. While these open-source models are designed with ethical alignment in mind, they lack the stringent restrictions of their closed-source counterparts. Meet Udeshi, a PhD student at NYU involved in the PromptLock project, stated, "These LLMs claim that they are ethically aligned—can we still misuse them for these purposes? And the answer turned out to be yes." It is plausible that covert PromptLock-style attacks have already been successful without detection, potentially leading to fully autonomous hacking systems. However, achieving this requires overcoming significant hurdles related to AI model reliability and inherent aversion to malicious use, all while evading detection.

Productivity Tools for Hackers

Current data on AI’s use in malicious activities largely comes from major AI companies themselves, and their findings are alarming. A November report from Google indicated that attackers were using AI tools, including Gemini, to dynamically alter malware behavior, enabling self-modification to evade detection. Google described this as ushering in "a new operational phase of AI abuse."

However, cybersecurity writer Kevin Beaumont pointed out that the five malware families analyzed in the Google report, including PromptLock, were easily detected and did not cause actual harm. He noted, "There’s nothing in the report to suggest orgs need to deviate from foundational security programmes—everything worked as it should."

Leonard acknowledges that this malware activity is in its nascent stages but believes public reporting of such findings is crucial for security vendors to develop more robust defenses against future, more dangerous AI attacks. He advocates for transparency, stating, "Sunlight is the best disinfectant… We want people to be able to know about this—we want other security vendors to know about this—so that they can continue to build their own detections."

Beyond developing new malware strains, attackers are also experimenting with AI to automate the hacking process. In November, Anthropic announced it had disrupted a large-scale cyberattack, the first reported instance executed with minimal human intervention. While details were scarce, a Chinese state-sponsored group reportedly used Anthropic’s Claude Code assistant to automate up to 90% of a "highly sophisticated espionage campaign."

"We’re entering an era where the barrier to sophisticated cyber operations has fundamentally lowered, and the pace of attacks will accelerate faster than many organizations are prepared for."
— Jacob Klein, head of threat intelligence at Anthropic

However, as with Google’s findings, there were caveats. Human operators selected targets and tasked Claude with identifying vulnerabilities, and only a handful of the 30 attempts were successful. The Anthropic report also noted that Claude "hallucinated" and fabricated data, overstating its findings, necessitating careful validation by the attackers. The report concluded, "This remains an obstacle to fully autonomous cyberattacks."

Gary McGraw, a veteran security expert, argues that existing security controls within most organizations would likely thwart such attacks, as the malicious execution, like vulnerability exploits, was not performed by the AI but by pre-existing automated tools that have been around for two decades. He states, "There’s nothing novel, creative, or interesting about this attack."

Despite these criticisms, Anthropic maintains that its findings signal a concerning shift. Jacob Klein emphasized that "Tying this many steps of an intrusion campaign together through [AI] agentic orchestration is unprecedented. It turns what has always been a labor-intensive process into something far more scalable."

Some remain unconvinced by the alarmist predictions. Hutchins suggests that AI hype has led to an overestimation of current AI capabilities within the cybersecurity industry. He believes the focus on "unstoppable AIs that can outmaneuver security" is not supported by current evidence, as AI capabilities do not yet meet the requirements for such advanced attacks.

BRIAN STAUFFER

Indeed, for the present, criminals primarily seem to be using AI to enhance their productivity. This includes using LLMs for writing malicious code and phishing lures, conducting reconnaissance, and for language translation. Jess observes this activity frequently, alongside efforts to sell AI-enhanced tools in underground criminal markets, such as phishing kits that track the success rates of various spam campaigns. While this activity exists in what could be termed the "AI slop landscape," she notes less evidence of "widespread adoption from highly technical actors."

However, unsophisticated attacks can still be highly effective. Models that produce "good enough" results enable attackers to target larger numbers of individuals than before. Liz James, a managing security consultant at NCC Group, explains, "We’re talking about someone who might be using a scattergun approach phishing a whole bunch of people with a model that, if it lands itself on a machine of interest that doesn’t have any defenses… can reasonably competently encrypt your hard drive. You’ve achieved your objective."

On the Defense

For now, researchers remain optimistic about the ability to defend against these emerging threats, regardless of their AI origins. Leonard states, "Especially on the malware side, a lot of the defenses and the capabilities and the best practices that we’ve recommended for the past 10-plus years—they all still apply." Standard security programs designed to detect conventional viruses and attack attempts remain effective, and many phishing emails are still intercepted by spam filters. These traditional defenses are largely expected to remain functional, at least for the foreseeable future.

In a notable development, AI itself is proving to be a powerful ally in cybersecurity, excelling at pattern recognition and correlation. Vasu Jakkal, corporate vice president of Microsoft Security, revealed that the company processes over 100 trillion signals flagged daily by its AI systems as potentially malicious or suspicious.

BRIAN STAUFFER

Despite the constantly shifting cybersecurity landscape, Jess finds encouragement in the open sharing of detailed information about attacker tactics among defenders. Initiatives like MITRE’s Adversarial Threat Landscape for Artificial-Intelligence Systems and the GenAI Security Project from the Open Worldwide Application Security Project are invaluable resources for understanding how criminals are integrating AI into their attacks and how AI systems themselves are being targeted. Jess notes, "We’ve got some really good resources out there for understanding how to protect your own internal AI toolings and understand the threat from AI toolings in the hands of cybercriminals."

PromptLock, the result of a limited university project, may not perfectly mirror real-world attacks. However, it underscored the significant technical capabilities of AI that should not be underestimated. NYU’s Udeshi expressed surprise at how easily AI managed a complete end-to-end attack chain, from system mapping and exploitation to crafting personalized ransom notes, achieving "80% to 90% success throughout the whole pipeline."

AI is evolving at a breakneck pace, with current systems already performing tasks that seemed fantastical just a few years ago. This rapid evolution makes it challenging to predict with absolute certainty what AI will or will not be capable of in the future. While researchers are confident that AI-driven attacks will increase in both volume and severity, their specific forms remain uncertain. The most extreme theoretical possibility involves an AI model capable of creating and automating its own zero-day exploits—highly dangerous attacks leveraging previously unknown software vulnerabilities. However, building and hosting such a model, while evading detection, would require billions of dollars, placing it within the realm of wealthy nation-states, according to Hutchins.

Engin Kirda, a professor at Northeastern University specializing in malware detection, believes such developments are already underway, particularly in China, given its advanced AI capabilities.

This prospect is undeniably alarming, but thankfully, it remains largely theoretical. A large-scale, effective, and clearly AI-driven campaign has yet to materialize. What is certain is that generative AI is already significantly lowering the bar for criminals. They will continue to experiment with new releases and updates, seeking novel ways to trick individuals into divulging sensitive information and parting with their money. For now, vigilance, carefulness, and consistent system updates are our most effective defenses.