A benchmark published to arXiv on October 14, 2026 has put a hard number on something developers have suspected for more than a year: AI coding agents are getting very good at writing code that works, and they are still bad at writing code that is safe to ship. The study, titled “Last Secure Code Benchmark: Is the Code You Vibe Secure?” and known informally as LSCBench, tested six frontier coding agents on 450 tasks drawn from 150 real-world vulnerabilities. The strongest agent passed as many as 91.3% of the functional tests. On the combined measure of “works and is secure,” that same leaderboard topper managed just 25.1%.
The gap between those two numbers is the story. It lands at a moment when AI coding agents are already producing roughly 30% more code without a matching increase in shipped software, and when enterprise security teams are rethinking how much autonomy to hand these tools. For a site that tracks the coding-tools beat closely, LSCBench is the clearest data point yet that “vibe coding” — writing software by describing what you want and letting an agent generate it — has a security tax that most teams have not priced in.
What the Last Secure Code Benchmark Actually Tested
According to coverage of the paper by Dev.to and the industry newsletter Agentry News, LSCBench was built by researchers affiliated with the National University of Singapore and Alibaba Group, including members of the Qwen team. The benchmark’s design choice is what makes it unusually revealing: task specifications handed to the coding agents asked only for working functionality. Security was never mentioned in the prompt. Hidden evaluation then checked two separate things — did the code do what was asked, and could the exact real-world exploit tied to that vulnerability still be replayed against the new code.
That replay step matters. Instead of relying on static analysis or pattern-matching for risky functions, LSCBench’s graders took 150 documented real vulnerabilities, stripped them down into 450 discrete coding tasks, and then fired the original attack at whatever the agent produced. If the agent’s code could still be exploited the same way the original flawed code was, it failed the security half of the test even if it passed every functional check.
The task set spans 24 categories from the Common Weakness Enumeration (CWE) list, the industry-standard taxonomy security teams use to classify software flaws. One source covering the paper, MindPattern, described the benchmark as spanning seven programming languages, naming Java, C, C++, JavaScript, Python, PHP and Go among them. That breadth is part of why the result traveled fast in developer circles: it is not a narrow test of one language or one type of bug, it is a cross-language stress test of whether “it compiles and passes the demo” is still a reasonable proxy for “it is safe to merge.”
Six Agents, One Very Wide Gap
Reporting on the paper consistently cites one name as the strongest performer: an agent identified as “Claude Fable 5,” which multiple outlets, including Dev.to and Agentry News, describe as reaching the top of both the functional leaderboard and the combined functional-and-secure leaderboard. Its 25.1% functional-and-secure score was the best in the field. The remaining five agents in the test — whose individual identities were not confirmed in the coverage available at publication time — scored between 16.4% and 24.7% on that same combined measure, according to the same reporting.
Put another way: even the best-performing coding agent in this benchmark shipped insecure code on roughly three out of every four tasks it technically completed. For the weakest of the six, that failure rate climbs past four in five. None of the six agents tested cleared even a third of the combined bar.
| Metric | Reported Figure |
|---|---|
| Benchmark name | Last Secure Code Benchmark (LSCBench) |
| arXiv identifier | 2610.14766v1 |
| Submission / update dates | Submitted Oct. 10, 2026; updated Oct. 14, 2026 |
| Total tasks | 450 |
| Source vulnerabilities | 150 real-world cases |
| CWE categories covered | 24 |
| Coding agents tested | 6 frontier agents |
| Functional pass-rate range (all 6) | 73.6% – 91.3% |
| Top agent: functional AND secure | 25.1% (“Claude Fable 5,” per secondary coverage) |
| Remaining five agents: functional AND secure | 16.4% – 24.7% |
Why Functional Correctness and Security Keep Diverging
The split LSCBench measured is not new in software engineering, but AI coding agents appear to have widened it rather than closed it. A model trained to satisfy a visible test suite will optimize for the thing it can see: does the function return the right value, does the endpoint respond, does the demo work. Security requirements are frequently implicit. Nobody writes “do not allow SQL injection” into a feature ticket, the same way nobody writes “do not let the building catch fire” into a construction contract. It is assumed. LSCBench’s entire design rests on exploiting that assumption — it never tells the agent to be secure, then checks whether the agent was secure anyway.
Illustratively, the kind of gap the benchmark is built to catch looks like this: an agent asked to “look up a user by the ID in the request” will often produce a functionally correct but injectable query, rather than the parameterized version a security-conscious engineer would default to.
-- Functionally correct, fails the security check
query = "SELECT * FROM users WHERE id = " + request.params.id
-- Functionally correct AND secure
query = "SELECT * FROM users WHERE id = ?"
db.execute(query, [request.params.id])Both snippets pass a basic functional test that simply checks whether the right user record comes back. Only the second survives the kind of attack-replay grading LSCBench applies. That one-line difference, repeated across hundreds of tasks and dozens of vulnerability classes, is effectively what separates a 91% functional score from a 25% functional-and-secure score.
The Vibe Coding Boom Behind the Numbers
LSCBench’s subtitle — “Is the Code You Vibe Secure?” — is a direct reference to “vibe coding,” the practice of describing a feature in plain language and letting an agent generate the implementation with minimal manual review. The approach has moved from hobbyist novelty to mainstream workflow faster than most prior developer-tooling trends. Tech-insider.org has previously covered vibe coding reaching game development through the Epic Games launcher and Google Play opening a vibe-coding path for app submissions that skips the traditional developer account process. Both moves lower the barrier to shipping AI-generated code, which raises the stakes on exactly the gap LSCBench just measured.
Separately, an October 2026 industry report citing JetBrains survey data found that 90% of developers surveyed are now using AI coding agents on at least a weekly basis. That adoption curve is the backdrop against which LSCBench’s numbers should be read: this is not a fringe concern about an experimental technology, it is a measurement of tools already embedded in the daily workflow of the large majority of professional developers.
What Developers Already Say About AI-Generated Code
LSCBench’s findings line up closely with sentiment developers have been reporting for over a year. According to the 2025 Stack Overflow Developer Survey, 81% of developers expressed concerns about the security and privacy of data when using AI tools. The same survey found that 46% of developers actively distrust the accuracy of AI tools, compared with just 33% who said they trust them, and 45% said debugging AI-generated code is more time-consuming than debugging code they wrote themselves. On adoption specifically, the survey recorded that 31% of developers reported currently using AI agents, while 38% said they had no plans to use them at all.
Those figures, drawn from Stack Overflow’s own published 2025 survey data, were already a warning sign before LSCBench put a specific percentage on the security shortfall. Now there is a benchmark number to attach to the distrust: even the most capable agent tested left three out of every four completed tasks exploitable.
| Data Point | Figure | Source |
|---|---|---|
| Developers concerned about AI tool security/privacy | 81% | Stack Overflow 2025 Developer Survey |
| Developers who actively distrust AI tool accuracy | 46% (vs. 33% who trust it) | Stack Overflow 2025 Developer Survey |
| Developers who say debugging AI code takes longer | 45% | Stack Overflow 2025 Developer Survey |
| Developers currently using AI agents | 31% (38% have no plans to) | Stack Overflow 2025 Developer Survey |
| Top agent’s functional-and-secure rate | 25.1% | LSCBench, arXiv 2610.14766v1 |
| Remaining five agents’ functional-and-secure range | 16.4% – 24.7% | LSCBench, arXiv 2610.14766v1 |
Market Impact: Pressure on Coding Agent Vendors
For vendors competing in the crowded coding-agent market, the LSCBench result changes the marketing conversation. Benchmarks like HumanEval and SWE-bench have spent years training the industry to treat “percent of tasks completed” as the main scoreboard, and tools have been sold on exactly that framing — including in tech-insider.org’s own coverage of how Claude Code, Codex, and GitHub Copilot compete for developer share. LSCBench argues that scoreboard is incomplete, and that a tool can look excellent on it while still generating code that fails a basic security replay test the majority of the time.
That argument gains weight next to separate, unrelated data on AI-assisted development load. Tech-insider.org has reported that developers now spend roughly 80% of their week reviewing AI-generated code, according to a BairesDev developer barometer survey. If review time is already consuming most of a developer’s week, and the underlying agent is only secure on a quarter of its completed tasks, the review burden is not a temporary growing pain — it is closer to a permanent cost of using these tools in their current form.
Historical Context: From Autocomplete to Autonomous Risk
The coding-assistant category has moved through roughly three phases in under five years. The first generation offered line-level autocomplete, suggesting the next few tokens based on context. The second generation introduced chat-based pair programming, where a developer could ask a model to explain, refactor, or generate a function and then decide whether to accept it. The current, third generation is agentic: tools that can read a codebase, plan multi-step changes, execute them, run tests, and iterate largely without a human reviewing every intermediate step.
Each phase reduced the amount of human judgment applied per line of code shipped. Autocomplete still required a developer to accept or reject every suggestion individually. Agentic workflows can produce an entire pull request’s worth of changes before a human looks at any of it. LSCBench’s finding is best understood as the security bill for that compounding reduction in per-line scrutiny: the less a human looks at each specific line, the more a benchmark like this one matters as a substitute check.
Competitive Landscape: How the Rest of the Industry Is Responding
LSCBench did not land in a vacuum. Several adjacent moves from major developer-tool makers in October 2026 point toward an industry that is already, separately, trying to put more guardrails around agent autonomy — even if none of them have yet published a security pass-rate as stark as LSCBench’s.
- Google’s Chrome for Developers blog detailed an October 2026 Chrome DevTools update built specifically for coding agents, adding an Agent Plugins 1.0 package, new security controls, performance tuning, and memory-analysis capabilities. The update includes a configurable
--no-javascript-evaluationoption that lets a developer block an agent from executing arbitrary JavaScript while it inspects a page. - Microsoft’s Visual Studio Code shipped version 1.142 on October 14, 2026, adding shared canvases for agent workflows, movable chat sessions that preserve context across moves, and — notably — expanded administrator controls over what permissions an agent is allowed to hold.
- GitHub has introduced push attribution in limited public preview for enterprise customers, adding authenticated pusher identity alongside existing author and committer metadata on commits, part of what the outlet DevOps.com described as an effort to rebuild Git’s underlying infrastructure to keep up with the scale of changes coding agents now generate.
None of these updates directly measure or guarantee secure output the way LSCBench’s grading does. But the direction is consistent: permissioning, attribution, and execution limits are all becoming first-class features in developer tools, which suggests vendors are reading the same signal LSCBench just quantified.
Security Tooling Built for This Exact Gap
The security side of the developer-tools market has already been building for a world where AI-generated code needs a second set of checks. Tech-insider.org has covered how AI gateway infrastructure like LiteLLM has shipped with its own exposed security flaws, affecting an estimated 9.6% of instances, and how secret-scanning tools like TruffleHog and Gitleaks catch leaked API keys that agents sometimes embed directly in generated code. There is also a growing category of tooling aimed squarely at defending AI agents from prompt injection, a related but distinct risk where the agent itself is manipulated into taking unsafe actions rather than simply producing a vulnerable artifact.
LSCBench adds a different, arguably more foundational data point to that stack: even absent any adversarial manipulation of the agent, and even when the agent is simply doing exactly what it was asked, the default output is insecure most of the time. That is a harder problem to patch with a gateway filter or a secret scanner alone, because the vulnerability is baked into the generated logic itself, not injected afterward.
Enterprise Governance Is Catching Up Slowly
Security and engineering leadership teams evaluating coding agents for enterprise rollouts have, until now, had very few independent, replay-tested benchmarks to point to. Most vendor-published numbers describe functional task completion, not the combined functional-and-secure metric LSCBench introduces. That gap in available data has made it difficult for a CISO or VP of engineering to compare agents on security grounds at all, as opposed to comparing them on raw completion rate or developer satisfaction, which is where most existing public benchmarks stop.
The Business Cost of Shipping Insecure AI-Generated Code
The business case against ignoring this gap is straightforward even without new breach data: a vulnerability that reaches production because an agent reproduced a known CWE pattern is, by definition, already understood by attackers, since LSCBench’s own tasks were built from previously documented real-world exploits. A security team discovering that a generated function matches a known weakness class is not dealing with a novel zero-day, it is dealing with a known failure mode that a generation pipeline reintroduced, at scale, across however many features that pipeline touched.
That reintroduction-at-scale dynamic is what distinguishes AI-generated vulnerabilities from traditional ones. A single engineer writing an injectable query is one mistake. An agent pattern that tends to generate injectable queries under a certain prompt structure is a mistake that can repeat across every team and every codebase using that agent, until either the agent is patched or every downstream team independently catches it in review.
What Security Teams Should Do With This Data Now
Treating LSCBench as a reason to ban coding agents outright would be an overcorrection given how deeply embedded these tools already are in daily workflows, per the JetBrains adoption figures cited above. A more proportionate response, consistent with how the industry has handled prior AI-tooling risk waves, is layering independent verification on top of agent output rather than trusting completion rate alone. That means treating a passing functional test as a necessary but not sufficient signal, and routing AI-generated code through the same CWE-aware review discipline that security teams already apply to code written by junior engineers.
It also means pressuring vendors for their own published functional-and-secure metrics, not just functional completion rates, the same way LSCBench’s authors structured their evaluation. Until vendors publish comparable numbers voluntarily, independent benchmarks like LSCBench are likely to remain the only source of this specific data point.
Predictions: Where AI Code Security Goes From Here
- Expect rival research groups to publish competing “functional-and-secure” benchmarks modeled on LSCBench’s attack-replay methodology within the next two to three academic cycles, since the approach is straightforward to replicate against new agents.
- Expect at least some coding-agent vendors to begin disclosing their own security pass rates voluntarily, likely framed favorably, once enough independent benchmarks exist that silence starts to look like an admission.
- Expect CWE-aware automated review to become a more prominent marketed feature of AI gateway and secret-scanning tooling, building on categories already covered around LiteLLM and TruffleHog-style scanning.
- Expect enterprise procurement processes for coding agents to start requesting benchmark citations similar to LSCBench as part of security questionnaires, mirroring how SOC 2 reports became a standard procurement ask for SaaS tools.
- Expect continued divergence between vendor marketing, which will likely keep emphasizing functional completion rate, and independent academic coverage, which is increasingly focused on the security half of that equation.
How This Compares to Prior Coding-Agent Benchmarks
Most widely cited coding-agent benchmarks to date, including those referenced in tech-insider.org’s own coverage of agent-versus-agent matchups, have measured a single axis: does the generated code pass the provided tests. LSCBench’s contribution is methodological as much as it is numerical — it is one of the more rigorous public attempts to measure a second axis, security, using the same real-world-grounded approach (attack replay against documented CWEs) rather than relying on static-analysis heuristics that can miss context-dependent vulnerabilities. That design choice is likely why the result traveled quickly through developer-focused outlets within 48 hours of the paper’s October 14 update, according to the spread of coverage across Dev.to, Agentry News, AIWeekly, and MindPattern.
FAQ: Last Secure Code Benchmark and AI Coding Agent Security
What is the Last Secure Code Benchmark (LSCBench)?
It is an academic benchmark, published to arXiv on October 14, 2026 under the title “Last Secure Code Benchmark: Is the Code You Vibe Secure?”, that tests whether AI coding agents produce code that is both functionally correct and free of known security vulnerabilities, using attack-replay grading against 150 real-world exploits.
How many coding agents were tested?
Six frontier coding agents were evaluated across 450 tasks. Secondary coverage identifies the top performer as an agent referred to as “Claude Fable 5,” though the full list of all six agent names was not confirmed in available reporting at publication time.
What was the headline result?
Functional correctness pass rates across the six agents ranged from 73.6% to 91.3%. On the combined measure of code that was both functional and secure, the best agent reached only 25.1%, while the remaining five scored between 16.4% and 24.7%.
Why does the benchmark not just use static analysis?
LSCBench’s grading replays the original real-world attack tied to each of its 150 source vulnerabilities against the agent’s generated code, rather than scanning for risky patterns. That approach is designed to confirm actual exploitability rather than flag theoretical risk.
Who built the benchmark?
According to coverage from Dev.to and Agentry News, the research team includes members affiliated with the National University of Singapore and Alibaba Group, including the Qwen team.
Does this mean developers should stop using AI coding agents?
The benchmark’s authors and the coverage around it frame the result as evidence that security review remains necessary rather than optional, not as an argument to abandon the tools. Given that an October 2026 report citing JetBrains data found 90% of surveyed developers already use AI coding agents weekly, the more realistic response is layering independent security review on top of agent output.
What programming languages did the benchmark cover?
One report covering the paper, from MindPattern, identified seven languages in scope: Java, C, C++, JavaScript, Python, PHP, and Go, spanning 24 CWE vulnerability categories.
How does this relate to “vibe coding”?
The benchmark’s subtitle directly references vibe coding, the practice of generating software from natural-language prompts with minimal manual review. LSCBench was designed to test exactly the scenario where a developer trusts an agent’s output without specifically prompting for security.