The PR came in on a Friday afternoon. Your AI coding agent had worked through the issue ticket, produced a clean diff, all CI checks were green, and the coverage report looked fine. Someone on the team hit merge before the weekend. On Monday morning, three customers reported that their exports were producing empty files.
The bug that was fixed? It was fixed. The three other things that broke? The agent never noticed them. It wasn’t trying to.
This is not a one-off story. This is becoming the dominant pattern in engineering teams that have adopted AI coding assistants and moved fast with them. A new benchmark from Alibaba researchers, published in March 2026, put a number on the pattern: 75% of AI coding models introduce regressions when asked to maintain a real codebase across months of evolution. Not once. Consistently, across 100 real repositories, spanning an average of 233 days and 71 consecutive commits per project.
The benchmark is called SWE-CI. And if you’re making decisions about how your team uses AI coding tools, it is worth understanding what it actually measures, and why the number it found is the one that matters.
What Existing Benchmarks Get Wrong
SWE-bench is the most cited evaluation for AI coding agents. The setup is clean: give the agent a GitHub issue and a snapshot of a repository, let it produce a fix, check whether the tests pass. If they do, the agent scores a point. If they don’t, it doesn’t.
That’s a reasonable test of whether an agent can write code. It is not a test of whether an agent can be trusted to maintain code.
The distinction matters because writing new code and maintaining existing code are fundamentally different jobs. Writing greenfield features is closer to translation: you have requirements, you produce an implementation, someone reviews it. Maintenance is closer to archaeology. You are working in a codebase that has history, accumulated assumptions, implicit contracts between modules, and tests that were written by people who understood context you may not have.
Real codebases evolve. A data model change you make today has consequences for a serialization function someone wrote eight months ago. An API endpoint you refactor can break a mobile client behavior that was never tested in the backend suite. The signal that a change is safe is not whether it passes today’s tests. It’s whether it keeps passing tests for the next 20 commits.
SWE-CI tests exactly that. Each of its 100 tasks plays out across up to 20 CI iterations, with the agent needing to make changes that pass current tests without breaking anything that was already working. The benchmark consumed over 10 billion tokens to run across 18 models from 8 providers. The result: most models introduced regressions on more than three quarters of tasks. Only Claude Opus managed to stay above a 50% zero-regression rate. Everyone else fell below 25%.
Why AI Agents Are Structural Local Optimizers
Understanding why the regression rate is so high requires understanding what an AI coding agent is actually doing when it touches your code.
It sees the failing tests. It traces the relevant code paths. It writes a change that makes those specific tests pass. Then it moves on.
What it is not doing is modeling how its change interacts with the 200 other tests in the suite that were already passing. It is not tracking the implicit contract between the function it just modified and the caching layer two modules away. It is not thinking about what the codebase looked like six months ago and what assumptions were baked into the test fixtures at that time.
This is not a prompt engineering problem. It is a structural property of how these systems reason. They are local optimizers. They fix the visible failure and optimize for the immediate evaluation signal, which is passing the failing tests. The EvoScore metric that SWE-CI introduced specifically weights later CI iterations more heavily than earlier ones, specifically to expose this pattern: agents that score well in the first few rounds but create cascading failures as the codebase evolves.
The researchers also found that AI-generated code is changing the character of codebases in measurable ways. Refactoring moves have dropped 70% since 2023. Code block duplication has risen 81%. These are not signs of a codebase getting healthier.
Maintenance work is estimated to consume between 60% and 80% of a software engineer’s time in production environments. The AI coding tools that have been most visibly productive have been most visibly productive on the remaining 20 to 40 percent. The part where you write something new, get it to pass tests, and merge it. Not the part where you keep an evolving system healthy over time.
What This Means for Your Team
If you are running an engineering team and you have adopted AI coding assistants, the SWE-CI findings are not a reason to throw those tools out. They are a reason to be honest about where those tools add value and where they shift risk to your team in ways that are easy to miss.
There are two failure modes I see most often.
The first is using AI coding agents for maintenance-heavy work without adjusting the review process. The agent produces a fix, the tests go green, someone approves the PR because the diff looks reasonable and nobody has time for anything more. Six weeks later, the bug that was fixed has returned in a slightly different form, and nobody knows why. The original fix introduced a regression that was dormant until a later change exposed it.
The second is treating benchmark scores as operational guidance. A tool that scores in the 80th percentile on SWE-bench is a good tool for writing new code in an isolated context. It is not necessarily a tool you can trust with a two-year-old financial processing module that seven engineers have touched and nobody has fully documented.
If you are evaluating AI-powered development options for your team, the practical questions to ask are narrower than “does this agent write good code.” They are: does this agent maintain regression discipline in a long-running codebase? What does our review process need to look like if the agent is doing 40% of the commits? And who owns the outcome when a regression ships?
That last question is the one that tends not to get asked until it needs to be.
Using AI Coding Agents Without Fooling Yourself
The teams I’ve seen use AI coding tools well share a few traits. None of them have stopped reviewing AI-generated code carefully just because the tests pass. Most of them have added explicit regression checks to their review process, not just test coverage, but targeted checks on the behavior of adjacent modules when a change touches something old.
Several have started categorizing work by type before deciding how much AI involvement makes sense. Greenfield features in a contained service? High AI involvement, lighter review overhead. A change to the payment processing pipeline, an authentication flow, or any module that multiple services depend on? Lower AI involvement, deeper human review, and a deliberate decision to keep a senior engineer in the loop on the implementation.
This is not about distrust. It is about matching the tool to the task. AI workflow automation works well when it is operating in bounded, well-defined contexts where the success condition is clear and the failure mode is visible. Maintaining a production codebase over time has neither of those properties by default. The success condition is diffuse (the system keeps working), and the failure mode is often invisible until it is not.
The SWE-CI benchmark is the first attempt to measure AI coding agents against the reality of what maintenance actually involves. The finding that most models introduce regressions on three-quarters of tasks is not a damning verdict on AI coding in general. It is a calibration signal. One you can build a smarter, more defensible process around.
The Part Nobody Mentions on the Vendor Slide
Every AI coding tool demo shows an agent writing a feature from a ticket. Clean diff, passing tests, merged in 90 seconds. Nobody shows the demo where the same agent quietly broke an edge case in the rate limiting logic that you will not discover for two months.
That is not because the vendors are being deceptive. It is because the demo context is the one the tools are best at. Greenfield, contained, immediate feedback. The maintenance context is harder to demo and harder to measure, and until SWE-CI came along, harder to benchmark.
Now there is a number. 75% regression rate, across 100 real codebases, maintained across months of real evolution. That number does not mean stop using the tools. It means use them with your eyes open.
The engineering teams that keep production software healthy over time are not the ones that adopt every new capability the moment it ships. They are the ones that understand where a capability adds value and where it introduces risk, and they build processes that reflect that understanding.
Your AI coding agent is a fast, talented junior who is very good at fixing the thing in front of them. You still need the senior engineer who knows what the thing in front of them connects to. That is not a limitation of the tool. It is a property of the work.
About This Article
This post examines the SWE-CI benchmark from Alibaba researchers, which tested 18 AI coding models against the actual job of software maintenance: evolving a real codebase across months of continuous commits without breaking what already works. The headline finding is 75% regression rates across most models. The reader comes away with a sharper frame for where AI coding agents add value, and where they quietly shift risk to your team.