AI Guardrails Are Blocking Legitimate Hackers, Pushing Researchers to Foreign Models

Strict guardrails imposed by leading AI companies to prevent malicious use of their models are inadvertently hindering the work of legitimate offensive cybersecurity researchers, forcing some to abandon frontier models for unregulated open-source alternatives, including those developed in China. The tension, highlighted by recent export controls on Anthropic’s Mythos model, reveals a fundamental conflict between AI safety measures and the needs of security professionals who proactively hunt for software vulnerabilities.

The double-edged sword of AI safety

For months, AI giants like Anthropic and OpenAI have implemented vetted access programs and strict guardrails designed to prevent their powerful models from being used to build or execute cyberattacks. However, these same restrictions are now being criticized by the very researchers they aim to protect. The core problem, experts say, is that the same AI capabilities that could help a malicious actor exploit a vulnerability are also essential for defenders trying to identify and fix those flaws before they are weaponized.

Chris Anley, chief scientist at NCC Group, a major security consulting firm, explained that asking an AI model to attempt to exploit a bug is a critical step in confirming a real vulnerability. When a guardrail causes the model to refuse such a request, it directly impedes defensive work. “This is where the whole offensive versus defensive and guardrails part comes in, because ‘fix this code’ as a prompt is both an essential mechanism for defense but also a roadmap for finding critical vulnerabilities in the code base,” Anley said. He compared the AI tool to a hammer: “You can’t build a house without a hammer. It’s definitely a tool but it’s also irreducibly a weapon as well.”

Vetted programs and inconsistent enforcement

Both Anthropic and OpenAI have established programs—such as OpenAI’s Trusted Access for Cyber and Anthropic’s Cyber Verification Program (CVP)—that allow vetted researchers to access models with fewer cybersecurity restrictions. Yet even within these programs, researchers report significant friction.

Chris Thompson, CEO of RemoteThreat and founder of the Offensive AI Con, said the guardrails are often inconsistent, changing how they work from day to day. “I think the practical impact is you spend a lot of time negotiating with the model instead of working on the core security program,” Thompson said. “Instead of analyzing a vulnerability and reasoning through the exploitability, you’re trying to find why you’re getting inconsistent results or why are models over-sanitizing the output.”

Paolo Stagno, CTO at CrowdFense, a company that develops and sells unknown vulnerabilities to government agencies, was more blunt, saying AI companies “essentially treat customers like children who need babysitting” with their vetted programs and guardrails. Stagno noted that he and his colleagues use frontier models only for reverse engineering, avoiding them for vulnerability discovery or exploit development to prevent sensitive data from being absorbed into future training runs. For critical work, they rely on open-source models run locally.

Pushing researchers toward unregulated models

A significant consequence of these restrictions, according to multiple researchers, is that legitimate cybersecurity professionals are being driven away from U.S.-governed AI systems toward foreign, unregulated alternatives. Thompson specifically pointed to Chinese open-source models like GLM, which can be downloaded and run locally with no vetting or usage restrictions. “You have these responsible researchers that are being pushed away from U.S.-governed systems to foreign-owned systems,” he said. “I think it’s more harmful than good to have these guardrails in place.”

This shift raises national security concerns, as researchers working on vulnerabilities critical to U.S. infrastructure may now rely on models developed by foreign entities with different data governance and security standards.

Not all researchers are affected

Not every offensive security expert feels impeded by guardrails. Giuseppe Cali, a security researcher specializing in zero-day discovery, said he doesn’t use AI for offensive work at all. He uses AI tools for initial reverse engineering and building supporting tools, but insists on discovering and weaponizing vulnerabilities himself. “I still want to own the actual bug discovery and weaponization myself and that wouldn’t change if all guardrails were lifted tomorrow,” Cali said. “I am jealous of my bugs, and I like this game too much to let models play it for me.”

However, for researchers whose employers are not part of the vetted programs, the situation is starkly different. One researcher at a smartphone-component manufacturer, speaking on condition of anonymity, said his company is not part of Anthropic’s CVP program, rendering its tools “barely useful” for vulnerability research because the guardrails are too strict. “If it catches wind we’re doing anything security related, it just stops and isn’t usable,” the person said.

Conclusion

The debate over AI guardrails in cybersecurity underscores a broader challenge: balancing the legitimate need to prevent AI-powered attacks with the equally critical need to equip defenders with the best possible tools. As Thompson warned, tightening restrictions further could leave defenders losing an accelerating arms race. “There’s this big storm coming. There’s this big wave of attacks that are going to happen at speed and scale like never before,” he said. “But the same security consulting firms and legit researchers that are trying to make a difference are being stifled right now.” Rather than more restrictions, Thompson called for AI labs to open up their programs, provide responsible access, and hold abusers accountable.

FAQs

Q1: Why are AI guardrails a problem for cybersecurity researchers?
AI guardrails are designed to prevent malicious use, but they also block legitimate tasks like asking a model to attempt to exploit a vulnerability, which is a standard step in confirming and fixing security flaws. This creates a barrier for defenders who need the same capabilities as attackers.

Q2: What are the alternatives for researchers blocked by guardrails?
Many researchers are turning to open-source AI models that can be run locally with no restrictions. This includes models from Chinese developers like GLM, which raises concerns about data security and national security when sensitive vulnerability research is conducted on foreign-controlled systems.

Q3: Do the vetted programs from OpenAI and Anthropic solve the problem?
Not entirely. Even within programs like OpenAI’s Trusted Access for Cyber and Anthropic’s Cyber Verification Program, researchers report inconsistent guardrail behavior and significant time wasted negotiating with models. Additionally, many legitimate researchers and companies are not included in these programs.