Type “cursor vs vscode” into Google and you get the same article forty times: a feature table, a pricing column, a verdict nobody asked for. None of it tells you whether Cursor’s Agent mode will actually fix the pagination bug that’s been sitting in your backlog for three weeks, or whether VS Code with GitHub Copilot will do it faster on your actual stack. Generic comparisons test toy problems on clean sample repos. Your codebase has fifteen years of inherited conventions, a flaky test suite, and quirks no blog post ever sees.
This tutorial skips the marketing claims and shows you how to build your own benchmark: a repeatable harness that runs the same real tasks from your own repository through Cursor and VS Code, scores the output against a rubric you define, and spits out a results table you can actually defend in a team meeting. By the end you’ll have 13 steps, a working scoring script, and a decision you made with your own data instead of someone else’s demo video.
Why Cursor vs VS Code Roundups Miss What Matters for Your Codebase
Cursor is a fork of VS Code, which is exactly why side-by-side comparisons are so misleading. Both editors share the same core, the same default keybindings, and largely the same extension API. Where they diverge is in how aggressively the AI layer is wired into the editing loop. Cursor built its product around Tab for multi-line edit predictions, Cmd+K for inline rewrites, and a Composer/Agent mode that can plan and execute changes across multiple files in one pass. VS Code, by contrast, ships Copilot as a layered extension with its own agent mode, chat panel, and workspace-aware context gathering.
The adoption numbers explain why this comparison keeps generating search traffic. “In 2025, 75.9% of developers reported using Visual Studio Code, while 17.9% reported using Cursor,” according to the 2025 Stack Overflow Developer Survey. That’s a massive incumbency gap, but it is not a quality gap — Cursor’s share has grown from a standing start in a market where VS Code has had a decade-long head start. The same survey found that 84% of developers said they were using or planning to use AI tools in their workflow, and 51% of professional developers reported using AI tools daily, Stack Overflow reported. Separately, IDC found that 90% of developers used a coding assistant to help build production software between summer 2023 and summer 2024, according to IDC’s Generative AI in the Software Development Life Cycle report.
With adoption at that level, the question for most teams isn’t “which editor is better” in the abstract. It’s “which one gets our specific tickets closed faster, with less rework, for less money.” That question only has one honest answer: run both on your own work and measure it. If you haven’t moved your settings and extensions over yet, our guide on migrating from VS Code to Cursor covers the one-click import flow first — you’ll want both editors fully configured before any of the steps below make sense.
There’s also a pricing dimension that generic comparisons tend to flatten into a single line item, and it matters more once you’re running a real benchmark than it does on a feature chart. Cursor sells its AI capability as the core product, with usage-based or plan-based pricing tiers tied directly to Agent and Composer calls. VS Code itself is free and open source; the cost sits entirely in whichever AI extension you bolt onto it, whether that’s GitHub Copilot’s subscription tiers or a different assistant entirely. That structural difference means a dollar-for-dollar cost comparison between the two is really a comparison between Cursor’s pricing plan and whatever AI extension you chose for VS Code — not between the editors themselves. Keep that distinction explicit in your own notes, because it changes how you interpret a cost gap in your final results table.
Why a Self-Run Benchmark Beats Public Leaderboards for This Decision
Public coding benchmarks like SWE-bench or Terminal-Bench are useful for comparing raw model capability, but they measure performance on curated, open-source GitHub issues that look nothing like your internal codebase. They don’t account for your team’s private conventions, your monorepo’s build quirks, your test suite’s flake rate, or the specific extensions and MCP servers your engineers actually run day to day. A model that tops a public leaderboard can still underperform badly on your proprietary code if it was never exposed to anything resembling your architecture during training or fine-tuning.
Vendor-published comparisons have the opposite problem: they’re optimized to showcase the vendor’s own strengths. A demo video showing Composer refactor a sample Express app in ninety seconds tells you that the demo was rehearsed, not that the same tool will refactor your fourteen-year-old Rails monolith in the same time. The benchmark in this tutorial sidesteps both failure modes by using your own tickets, your own test suite, and your own pass/fail criteria. It will never scale to the sample sizes of a published leaderboard, but for the narrow question of “which tool should our team standardize on,” a six-task internal benchmark is more decision-relevant than a thousand-task public one.
What You’ll Build: A Reproducible AI Editor Benchmark
By the end of this tutorial you will have a small, version-controlled benchmark kit that lives alongside your project: a rubric file describing what “good” looks like for each task category, a pair of git worktrees that give each editor an identical, isolated copy of your repo, a shell script that times each run and captures the diff, and a scoring script that turns raw logs into a results table. None of this requires exotic tooling — bash, git, Python 3, and the two editors you’re already deciding between.
The output is not a single number declaring a winner. It’s a per-task, per-category breakdown: Cursor might close bug-fix tickets faster while VS Code with Copilot produces cleaner diffs on large refactors, or the reverse. That granularity is the entire point — “best overall” verdicts are what generic articles sell you, and they’re exactly what this harness is built to replace.
Prerequisites and Versions You’ll Need
- Cursor, latest stable release — check Settings → General → About for your installed build before starting, since the import UI and Agent mode behavior have shifted release to release
- VS Code, latest stable release, with GitHub Copilot (or another AI extension you want to test) signed in and active
- Git 2.40 or newer, with
git worktreesupport - Python 3.10 or newer, for the scoring script
- Bash 4 or newer (macOS users on the stock Bash 3.2 should install a newer version via Homebrew)
- jq, for quick JSON inspection from the terminal (optional but convenient)
- A real repository with a working test suite — your own project, not a toy sample app
- At least 2 GB of free disk space per worktree copy, more for large monorepos
One housekeeping note before you start: Cursor installs extensions from the Open VSX registry rather than the Microsoft Marketplace, a transition that Learn Cursor’s migration documentation dates to June 2025. If your test tasks depend on a VS Code-only extension — Microsoft’s Remote-SSH or Live Share are common examples — confirm it has an Open VSX equivalent before you commit to a task list, or you’ll be benchmarking two different development environments instead of two editors.
Step 1: Install and Version-Lock Both Editors
Update both editors to their current stable release before you touch the harness. Benchmarking an outdated Cursor build against the newest VS Code release (or vice versa) invalidates every result that follows — you’d be measuring a version gap, not a product gap. Open each editor’s About dialog, write the version string into a plain text file called versions.txt inside your benchmark folder, and commit it. This single file is what lets you re-run the same benchmark in three months and know immediately whether a result changed because the tools changed or because something in your methodology drifted.
While you’re in settings, disable any auto-update behavior on both editors for the duration of the benchmark. An editor that updates itself mid-run, silently changing model routing or default context window size, will quietly poison your data without any error message telling you it happened.
Step 2: Pick a Representative Task Set From Your Own Repo
This is the step most people skip, and it’s the one that makes or breaks the whole exercise. Pull five to eight real, recently closed tickets from your issue tracker — not toy examples, not “write a function that reverses a string.” Spread them across at least three categories so the results reflect how your team actually works:
- Bug fixes — a ticket with a clear repro and an existing failing test, or one you can write a failing test for
- Small feature additions — a self-contained new capability that touches two to five files
- Refactors — a change that should not alter behavior, verifiable entirely by the existing test suite staying green
For each task, write down the acceptance criteria in plain language before you run anything. If you can’t describe when a task is “done” in one sentence, you can’t score it objectively later, and the whole benchmark degrades into gut feeling with extra steps.
A concrete example helps here. A real bug-fix candidate might read: “Pagination on the /orders endpoint returns duplicate rows when page size exceeds 50. Repro: hit GET /orders?page=2&size=75 against the seeded test database and compare the returned IDs to the expected fixture list in orders_fixture.json.” That’s specific enough that both editors receive identical information, specific enough that “done” means the fixture comparison passes, and realistic enough that it actually touches your pagination logic rather than a sanitized textbook version of the same bug. Avoid tickets that depend on tribal knowledge not captured anywhere in the ticket itself — if a task requires a Slack thread from four months ago to understand, paste the relevant context into the ticket description so both runs have equal access to it.
Step 3: Create Isolated Git Worktrees for a Fair Comparison
Each editor needs its own clean, identical copy of the repo at the same commit, so neither run can accidentally benefit from cached state, stale dependencies, or leftover edits from the other tool. Git worktrees solve this without the overhead of cloning the repo twice:
# From your main repo, at the commit you want to benchmark against
git fetch origin
git worktree add ../bench-cursor -b bench-cursor-run1 main
git worktree add ../bench-vscode -b bench-vscode-run1 main
# Install dependencies identically in both
cd ../bench-cursor && npm ci
cd ../bench-vscode && npm ciOpen ../bench-cursor in Cursor and ../bench-vscode in VS Code. Keep both editors closed between tasks and reopen fresh for each one — a warmed-up index or a chat history from a previous task can leak context that one tool had and the other didn’t, which breaks the comparison just as badly as mismatched dependencies would.
Step 4: Define a Scoring Rubric Before You Run Anything
Write the rubric before you see a single result. This is the step that keeps confirmation bias out of the process — it’s tempting to decide a tool “felt faster” after the fact and then build a rubric that justifies it. Lock the weights in first.
{
"task_id": "fix-pagination-bug",
"category": "bug-fix",
"pass_criteria": {
"tests_pass": true,
"max_files_touched": 3,
"no_unrelated_diff": true
},
"score_weights": {
"correctness": 0.50,
"time_to_green": 0.25,
"diff_size": 0.15,
"cost": 0.10
}
}Save one of these JSON files per task in a rubrics/ folder. Correctness should always carry the heaviest weight — a fast wrong answer is worse than a slow correct one, and any benchmark that doesn’t reflect that will reward the wrong tool.
The exact weights should shift depending on what your team actually optimizes for. A startup burning through runway might weight cost and speed more heavily because shipping velocity is the scarce resource. A team maintaining a regulated financial system might weight correctness and diff size even more heavily than the example above, since an unreviewable 40-file diff creates audit risk regardless of whether the tests pass. Don’t copy the weights from this article verbatim — use them as a starting template and adjust based on a five-minute conversation with your tech lead about what actually matters for your next two quarters.
| Criterion | Typical weight | How it’s measured | Tool used |
|---|---|---|---|
| Correctness | 40–50% | Existing test suite passes; new tests for the bug pass | CI runner / local test command |
| Time to green | 20–25% | Wall-clock minutes from task start to passing tests | Stopwatch script (Step 7) |
| Diff size | 10–15% | Lines and files changed relative to a minimal human fix | git diff --stat |
| Token/dollar cost | 10–15% | Usage reported in the editor’s own cost dashboard | Cursor Usage panel / Copilot billing page |
| Rework needed | 5–10% | Number of follow-up prompts before acceptance criteria are met | Manual tally during the run |
Step 5: Configure Cursor’s Agent and Composer Mode
Open Cursor Settings with Cmd+Shift+J on macOS or Ctrl+Shift+J on Windows and Linux. Confirm Composer/Agent mode is enabled and set to the model tier you intend to keep for the entire benchmark — switching models between tasks is one of the most common ways people accidentally invalidate their own results. If your team’s actual workflow leans on Tab completions and Cmd+K rewrites rather than full Agent runs for smaller tasks, decide that mapping now and apply it consistently: bug fixes might go through Cmd+K, while multi-file features go through Composer.
If your task set depends on project-specific conventions, write them into a Cursor rules file so the agent has the same guardrails a human teammate would. Our walkthrough on setting up Cursor alongside Claude Code covers how to structure that context file if you’re also testing a terminal-based agent as a third data point.
Step 6: Configure VS Code’s AI Assistant
In the VS Code worktree, confirm Copilot’s agent mode is signed in and pointed at the model tier equivalent to what you picked for Cursor. This is the single hardest part of the whole exercise to get genuinely fair: Cursor and Copilot don’t expose identical model names or pricing tiers, so an exact match is rarely possible. The honest approach is to document exactly which model each tool used for the run and note the price-per-task alongside the results table, rather than pretending the comparison is apples-to-apples when it isn’t.
If you’re running MCP servers in either editor to extend context — a linked ticket tracker, a database schema reader, a design-doc search — make sure the same servers are configured on both sides before you start. Our guide to setting up Cursor MCP servers alongside VS Code agents walks through keeping that layer in parity so neither tool has a context advantage the other lacks.
Step 7: Build the Benchmark Harness Script
The harness is deliberately low-tech: it starts a timer, waits for you to signal that the AI run is complete, stops the clock, captures the diff stat, and runs your test command. Save this as run_task.sh in your benchmark folder and make it executable with chmod +x run_task.sh.
#!/usr/bin/env bash
# run_task.sh
LABEL=$1
REPO_PATH=$2
TEST_CMD=$3
echo "Starting timer for $LABEL. Begin the task in the editor now."
START=$(date +%s)
read -r -p "Press Enter once the agent reports the task is done: " _
END=$(date +%s)
cd "$REPO_PATH" || exit 1
git diff --stat > "/tmp/${LABEL}_diffstat.txt"
eval "$TEST_CMD" > "/tmp/${LABEL}_test_output.txt" 2>&1
TEST_EXIT=$?
echo "${LABEL},$((END-START)),${TEST_EXIT}" >> /tmp/benchmark_log.csv
echo "Logged: $LABEL took $((END-START))s, test exit code $TEST_EXIT"Run it once per editor, per task: ./run_task.sh cursor-bugfix01 ../bench-cursor "npm test" and ./run_task.sh vscode-bugfix01 ../bench-vscode "npm test". The manual Enter-key step is intentional — fully automating agent completion detection is fragile across tools with different “done” signals, and a human watching the session also catches qualitative issues (an agent looping, rewriting unrelated files) that a pure timer would miss entirely.
Step 8: Run Your First Task in Both Editors
Give the AI the exact same prompt, worded identically, in both editors. Copy-paste it rather than retyping — small wording differences between runs introduce a variable you didn’t intend to test. For a bug fix, that prompt should be the ticket description plus the repro steps, nothing more. Resist the urge to add hints to one editor that you didn’t give the other, even if you think you know what’s wrong; that’s the operator bias covered in the pitfalls section below.
Let the agent run to completion without interrupting it, even if it looks like it’s going down the wrong path. If it genuinely gets stuck in a loop, note that as a result in itself — “never produced a passing fix after 10 minutes” is valid, useful data, not a run to discard.
Step 9: Score the Output Against Your Rubric
Once the harness logs a run, check the diff stat and test output files against the rubric’s pass criteria. A typical /tmp/cursor-bugfix01_diffstat.txt looks like this:
src/pagination/pageSize.ts | 14 +++++++-------
src/pagination/pageSize.test.ts | 22 ++++++++++++++++++++++
2 files changed, 25 insertions(+), 11 deletions(-)Against a rubric capping “max_files_touched” at 3, that run passes the diff-size criterion. Cross-reference the test output file for a clean pass — exit code 0 — and you have everything you need to fill in one row of the results table. Do this scoring pass immediately after each task while the context is fresh; batching scoring to the end of a long session invites you to misremember which run had which quirks.
Step 10: Track Token and Dollar Cost Per Task
Cost doesn’t show up in a diff stat, so it has to be recorded separately. Cursor exposes a usage dashboard under its account settings that breaks down spend by request; Copilot’s usage shows up on GitHub’s billing page at the organization or individual level. Neither updates instantly, so check it a few minutes after each task rather than immediately, and record the delta in dollars (or included-plan “requests,” if you’re on a flat-rate tier) in a simple CSV alongside your timing log:
editor,task_id,cost_usd
cursor-agent,fix-pagination-bug,0.42
vscode-copilot,fix-pagination-bug,0.31If you’re on a flat-rate plan for either tool, note that explicitly instead of writing “0” — a flat $20/month plan that happens to absorb a task’s cost is a different result than a tool that is genuinely cheaper per task, and your results table should make that distinction legible rather than hiding it behind a zero.
Step 11: Aggregate Results Into a Scoring Table
With the CSV logs in place, a short Python script turns raw numbers into something you can read at a glance:
import csv, json
from pathlib import Path
rows = []
with open("/tmp/benchmark_log.csv") as f:
for label, seconds, exit_code in csv.reader(f):
diffstat = Path(f"/tmp/{label}_diffstat.txt").read_text()
rows.append({
"run": label,
"seconds": int(seconds),
"passed": exit_code == "0",
"files_changed": diffstat.count(" | "),
})
print(json.dumps(rows, indent=2))Running it produces output you can paste straight into a team wiki page or pull request description:
[
{
"run": "cursor-bugfix01",
"seconds": 318,
"passed": true,
"files_changed": 2
},
{
"run": "vscode-bugfix01",
"seconds": 402,
"passed": true,
"files_changed": 3
}
]From there, build a summary table across every task category you tested. A sample from a four-task run might look like this:
| Task | Editor | Time to green | Files changed | Cost | Passed |
|---|---|---|---|---|---|
| Pagination bug fix | Cursor Agent | 5m 18s | 2 | $0.42 | Yes |
| Pagination bug fix | VS Code + Copilot | 6m 42s | 3 | $0.31 | Yes |
| Add CSV export feature | Cursor Agent | 14m 02s | 5 | $1.15 | Yes |
| Add CSV export feature | VS Code + Copilot | 11m 50s | 4 | $0.89 | Yes |
| Refactor auth middleware | Cursor Agent | 21m 40s | 9 | $1.80 | No (2 tests broke) |
| Refactor auth middleware | VS Code + Copilot | 26m 15s | 7 | $1.42 | Yes |
This is a fabricated example to illustrate the shape of the output — your own numbers will depend entirely on your codebase, your rubric, and the models you chose in Steps 5 and 6. Note the pattern a real benchmark reveals: neither tool wins every category, which is exactly the nuance a generic “cursor vs vscode” listicle can’t give you.
Interpreting Results: How Big a Gap Actually Matters
A six-task benchmark is not a statistically rigorous sample, and treating a 12% time difference as proof of anything is a mistake in the opposite direction from trusting a vendor’s demo video. Before you act on a gap, ask whether it shows up consistently across multiple tasks in the same category, or whether it’s driven by one outlier run where an agent got lucky or unlucky. A single task where Cursor finished eight minutes faster doesn’t mean much on its own. The same gap showing up across four separate bug-fix tasks, with the same editor finishing faster every time, is a real signal worth acting on.
Use this rough guide when deciding whether a result is worth changing your team’s workflow over:
| Observed gap | Sample size | What it suggests | Recommended action |
|---|---|---|---|
| Under 10% | Any | Likely noise — within normal run-to-run variance | Don’t act yet; re-run the same task once more |
| 10–25% | 1–2 tasks in a category | Could be real, could be a lucky or unlucky run | Run at least two more tasks in that category before deciding |
| 10–25% | 3+ tasks, consistent direction | A genuine, if modest, advantage | Weight it in your decision, but don’t treat it as decisive alone |
| Over 25% | 3+ tasks, consistent direction | A real, team-relevant difference | Strong candidate to drive a workflow or tooling decision |
| A hard fail (tests broke) | Any | Correctness problem, not a speed tradeoff | Disqualify that run from the category regardless of time saved |
Treat a correctness failure as disqualifying rather than something to average into a composite score. A tool that finishes a refactor in eighteen minutes but breaks two tests has not actually finished the task faster than one that takes twenty-six minutes and passes — it has produced an incomplete result more quickly, which is a different thing entirely. Your rubric’s weighting in Step 4 should already reflect this, but it’s worth restating when you’re staring at a results table and tempted to average away an inconvenient fail.
Step 12: Repeat Across Task Categories
One task per category tells you almost nothing — a single bug fix could go well or badly for reasons that have nothing to do with the tool. Run at least two to three tasks per category, and if a result looks surprising or inconsistent with the others, run it a second time before you trust it. Agents are not deterministic; the same prompt against the same codebase can produce a different approach on a second attempt, and a genuine outlier is only visible once you’ve seen the normal spread.
Keep a running log across the whole benchmark, not just a table per task. A simple append-only markdown file with one dated entry per run, including anything that felt off during the session, becomes invaluable when you’re explaining a surprising result to your team three weeks later and can’t remember the details.
Step 13: Make the Call and Document It for Your Team
Once you have results across every category, write the decision down with the reasoning attached, not just the conclusion. A note like “we’re standardizing on Cursor for feature work because it averaged 20% faster time-to-green on our last six feature tickets, but keeping VS Code with Copilot available for refactors where it produced smaller diffs” is a decision your team can revisit and argue with later. “We picked Cursor because it’s better” is not — it gives the next engineer nothing to push back against if their experience differs, and no way to tell whether the decision is still correct a year from now when both tools have shipped several updates.
Commit the rubric files, the harness scripts, and the final results table to a benchmarks/ folder in your repo. The next time either editor ships a major change to its agent behavior — and both ship those changes often — you can re-run the exact same tasks and get a like-for-like comparison instead of starting from scratch.
Common Pitfalls When Benchmarking AI Code Editors
- Testing toy problems instead of real tickets. A benchmark built on “reverse a linked list” tells you nothing about how either tool handles your actual conventions, your actual test flakiness, or your actual file layout.
- Not resetting state between runs. Stale
node_modules, a cached test runner, or leftover build artifacts from a previous task can make one editor look faster for reasons that have nothing to do with the AI. - Comparing mismatched model tiers. Running Cursor on its fastest, cheapest model against Copilot on its most capable model (or vice versa) measures a pricing-tier decision, not an editor decision.
- Operator bias. It’s very easy to subconsciously nudge the tool you prefer with a slightly better follow-up prompt. Write every prompt down verbatim before you start and reuse it exactly.
- Ignoring cost entirely. A tool that’s 15% faster but three times more expensive per task isn’t automatically the winner — let your rubric’s cost weight do the deciding, not your gut.
- Running a single trial per task. Agentic coding tools are non-deterministic. One run is an anecdote; three runs is data.
- Letting extension gaps distort the test. If a task depends on a VS Code-only extension that isn’t available on Cursor’s Open VSX registry, you’re benchmarking two different environments, not two editors.
Troubleshooting Guide
Cursor’s Agent mode keeps editing the same file in a loop. Interrupt it, note the loop as a failed run in your log, and restart with a more specific prompt describing the expected file boundary. Don’t silently retry and only keep the “good” attempt — that’s survivorship bias baked into your own data.
Copilot’s agent mode option is greyed out in VS Code. Confirm your GitHub account has an active Copilot subscription and that the extension has fully reloaded after sign-in; a stale extension host is the most common cause.
Git reports “fatal: already checked out” when adding a worktree. You’re trying to check out a branch that’s already active in another worktree. Create a dedicated benchmark branch for each worktree instead of reusing main directly, as shown in Step 3.
The diff stat shows far more files changed than expected. Check whether one editor auto-formatted the entire file on save while the other didn’t. Disable format-on-save in both editors for the duration of the benchmark, or the diff-size criterion becomes meaningless.
Tests fail in both editors on the same task. Before blaming either tool, check whether the test suite itself is flaky by running it twice on the unmodified baseline. A test that fails intermittently on its own will fail intermittently no matter which editor touched the code.
run_task.sh exits immediately with “permission denied.” You forgot chmod +x run_task.sh after creating the file, or you’re invoking it without the leading ./.
The timer logs a negative or near-zero duration. This happens if you press Enter before the agent has actually finished — the script trusts you, not the editor, to know when a run is complete. Watch the editor’s status indicator rather than estimating.
Cost figures in Cursor’s dashboard don’t match what you expect. Usage dashboards on both platforms can lag by several minutes. Wait, refresh, and record the delta rather than the running total, since the running total includes everything since your last billing cycle reset.
A needed extension isn’t available in Cursor. Check whether the publisher offers a standalone .vsix package you can sideload, which Rapid Dev’s guide to managing Cursor extensions covers as a workaround for most non-Microsoft-exclusive tools missing from Open VSX. If it’s a Microsoft-exclusive extension with no workaround, substitute an equivalent task that doesn’t depend on it rather than forcing an unfair test.
Advanced Tips: CI Integration, Blind Scoring, and Multi-Model Runs
Once the manual version of this harness is working, a few upgrades make it repeatable at team scale rather than a one-off exercise you ran once and filed away.
Archive results as CI artifacts. Even though the agent runs themselves stay manual (agents aren’t yet reliable enough to drive unattended in most CI environments), you can automate the boring part — storing and comparing results over time — with a lightweight workflow:
name: ai-editor-benchmark
on: workflow_dispatch
jobs:
record-results:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Upload benchmark log
uses: actions/upload-artifact@v4
with:
name: benchmark-results-${{ github.run_id }}
path: /tmp/benchmark_log.csvBlind-score the diffs. If your team is worried about bias creeping into the correctness scoring, strip the editor label off each diff before a second reviewer scores it for code quality. Pass/fail on tests is objective, but “is this the right abstraction” is not, and blind review catches a reviewer who unconsciously favors whichever tool they personally prefer.
Test more than two tools in parallel. The same worktree-and-harness pattern extends cleanly to a three- or four-way comparison if your team is also evaluating a terminal-based coding agent. If you’re weighing Cursor or VS Code against something like Claude Code or Codex CLI, our breakdowns of Claude Code vs Codex vs Copilot and GitHub Copilot vs Cursor vs Claude Code are a useful starting point for which additional tool is worth adding to your own benchmark rather than trusting their published numbers outright.
Re-run quarterly, not once. Both Cursor’s Agent/Composer stack and VS Code’s Copilot agent mode change frequently enough that a benchmark from six months ago is stale. Treat the benchmarks/ folder as a living artifact, not a one-time report, and schedule a re-run whenever either vendor ships a headline change to their agent behavior.
Rotate which engineer runs the benchmark. A single person running every task introduces their own prompting habits as a hidden variable, even with verbatim prompts — how quickly someone notices an agent going off track and interrupts it still varies by operator. If more than one engineer can rotate through running tasks, do it, and note who ran which task in your results log. If a particular engineer’s runs consistently favor one tool, that’s worth investigating before you trust the aggregate numbers.
Separate “time to first attempt” from “time to accepted fix.” Some tasks will pass on the first try; others take two or three rounds of follow-up prompting before the agent lands on something acceptable. Logging both numbers separately — rather than only the final accepted time — tells you something a single timestamp can’t: whether a tool tends to nail tasks in one pass or needs iterative correction to get there, which matters a great deal for how much hands-on attention a team should expect to give it day to day.
The Complete Working Project
Pulled together, the folder structure for the full benchmark kit looks like this:
benchmarks/
versions.txt # editor build numbers, locked at test start
rubrics/
fix-pagination-bug.json
add-csv-export.json
refactor-auth-middleware.json
run_task.sh # timer + diff capture + test runner
score.py # aggregates benchmark_log.csv into JSON
results/
2026-10-run1.md # dated results table + notes
prompts/
fix-pagination-bug.txt # exact prompt text, reused verbatimEverything in that folder is plain text, which is deliberate: no proprietary benchmark format, nothing that locks you into a vendor’s own measurement tooling, and nothing that stops a teammate from auditing exactly how a result was produced six months after the fact. Commit it next to the code it was tested against, and the next engineer who wants to relitigate “should we switch editors” inherits a working harness instead of starting the argument from zero.
Frequently Asked Questions
Is Cursor actually built on VS Code, or just similar to it?
Cursor is a fork of VS Code’s open-source codebase, which is why it shares the same default keybindings, largely the same extension API, and a similar settings format. The divergence is in how deeply the AI layer is integrated into the core editing loop rather than bolted on as an extension.
Can I run this benchmark without a dedicated test suite?
You can, but your correctness scoring becomes manual and more subjective. If your project has little or no automated testing, consider writing a handful of quick smoke tests for just the files involved in your benchmark tasks before you start — it will make the “passed” column meaningfully more trustworthy.
How many tasks do I need for a result I can trust?
Two to three tasks per category (bug fix, feature, refactor) is a reasonable floor for a first pass. If a category shows a split or surprising result, add more tasks in that category specifically rather than padding the other categories that already look consistent.
Does this work if my team already uses both editors for different projects?
Yes — in fact that’s one of the more common outcomes of running this kind of benchmark. Many teams land on a split decision, using one editor for a specific project type or task category rather than forcing a single company-wide standard.
What if Cursor and Copilot can’t be put on equivalent model tiers?
Document the mismatch explicitly in your results table rather than pretending it doesn’t exist. A note like “Cursor ran on its standard tier; Copilot ran on its equivalent-priced tier, not an exact model match” preserves the honesty of the comparison for whoever reads the results later.
How often should we re-run the benchmark?
Quarterly is a reasonable cadence for most teams, or immediately after either vendor ships a headline change to their agent behavior. Both tools iterate on their AI features often enough that a benchmark older than six months should be treated as outdated.
Can I use this harness to compare more than two editors?
Yes. The worktree-and-rubric pattern isn’t specific to Cursor and VS Code — add a third worktree and a third run_task.sh invocation per task for any additional tool, whether that’s a different AI extension inside VS Code or a separate terminal-based agent entirely.
What happens if one editor refuses the task entirely, say for a security-sensitive file?
Log it as a hard fail for that task rather than excluding it from the results. A tool that declines to touch certain code due to safety guardrails is giving you real information about how it will behave in your actual day-to-day workflow, and silently dropping that data point hides a limitation your team needs to know about before standardizing on it.