A study reveals that AI Code assistants may slow down experienced developers. The METR (Model Evaluation & Threat Research) group has released their findings on the subject. The result challenges the popular belief regarding benefits of AI code assistants.
Advertisеment
Researchers observed that experienced developers using AI assistants took 19% longer to complete tasks compared to those working without AI support. Despite this, participants believed AI accelerated their work by 20%. Prior to starting the tasks, they had expected a 24% improvement in speed. Expert predictions from economists and machine learning specialists were even more optimistic, forecasting time savings of 39% and 38%, respectively.
The experiment involved 16 open source developers with moderate experience using AI tools. Participants solved 246 tasks such as bug fixes and new feature additions. Their task where coming from real GitHub issues in projects with which the developers had at least five years of experience. Some tasks required manual coding, while others allowed use of any AI assistant. Most developers picked the Cursor code editor with the Claude 3.5/3.7 Sonnet model.
The study ran from February to June 2025 and included well-known open source projects such as mito, stdlib, ghc, cabal, flair, jsdom, hypothesis, trieve, scikit-learn, gpt-neox, and transformers. On average, these projects featured 23,000 GitHub stars, 1.1 million lines of code, 20,000 commits, and 710 contributors.
Several factors may explain the slower performance when using AI
- Low quality of AI recommendations. Developers accepted less than 44% of AI-generated suggestions and spent significant time reviewing and correcting them.
- Overconfidence in AI capabilities. Excessive optimism led to inflated expectations about tool performance.
- Good developer expertise. Participants' deep familiarity with the codebases made AI assistance notable useless.
- Complexity of the repositories. AI tools performed poorly on large, complicated codebases.
- Lack of contextual understanding by AI. AI systems failed to recognize the exact context of the code they modified.
While developers spent less time writing code, searching for information, or reading documentation, these gains disappeared due to increased time spent generating prompts, waiting for responses, analyzing outputs, and revising results. Instead of focusing on core development, developers wasted their time talking with AI, verifying and refining its output.
The researchers suggest that the results may be different if the experiment involved less experienced developers or unfamiliar codebases. In such cases, performance metrics may improve, especially if developers have more trust in AI-generated code.
Another Survey Reveals Mixed Perceptions of AI Use
A separate survey conducted by Qodo of 609 developers who use AI in their workflows revealed mixed perceptions. 78% reported increased productivity after adopting AI tools. 60% claimed improved overall code quality, while 20% noted a decline in quality.
Despite perceived benefits, 76% of respondents claimed that they do not merge AI-generated code without manual review. Many developers reported feeling more productive individually, as they produced more code faster. However, any of the AI-assisted contributions required the developers to spend additional time on reviewing the code.
Support us
Winaero greatly relies on your support. You can help the site keep bringing you interesting and useful content and software by using these options:
