Benchmark

Eight prompts a developer would actually type, answered three times against the same Next.js application — once the way an agent searches a codebase today, once on this server’s free tier, once on Pro. Every step was genuinely executed and its real output measured. Run 2026-09-14.

The project is dubinc/dub, open source and 3,593 source files. Deliberately not our own code and not a customer’s: you can clone it and check every number on this page yourself, which is the only kind of benchmark worth publishing.

The question it answers

Not “is this agent smarter” — that is not measurable in a way anyone should trust. The question is narrower and answerable: how much context has to enter the model before the file that owns a task is on the table? That cost is paid on every prompt, by every agent, before any work begins, and it is the cost this server exists to remove.

The project

dub is a link-management product: 3,593 TypeScript and TSX files, Next.js App Router, several domains routed by host, and a large enterprise subtree. It is not a toy repository — on a fifty-file project no discovery strategy costs anything, and the difference only appears at the size where developers actually want help.

It is also a hard case for us, chosen knowing that. Everything in it is named in plain English, so grep works about as well as grep can work, and it has no translation files, so this server’s ability to learn a project’s domain vocabulary from its own i18n never comes into play. The gap you see here is the gap on a codebase that suits the other side.

How each side was run

Searching for it. Four steps, in the order an agent takes them: glob for the nouns in the request; grep the developer’s own phrasing; when that phrase returns nothing — which is usual, because people describe behaviour and code names things — grep the words separately; then read the 3 most promising files. Candidates are ranked by how much of the request their path echoes, the way a person decides what to open.

That ranking step matters, and it was added deliberately: reading raw grep hits in the order they came out produced a much weaker baseline and a much more flattering chart. A comparison is only publishable if the other side was given its best shot. With it, searching finds the right file in 6 of 8 and this server in 6 of 8 — a tie, and not the same tasks either way. The claim here is about cost, not accuracy.

One answer key was widened after the fact and it is worth saying so. For the date-range task both sides returned the same date-range picker component, which the key had not listed; that was the key being wrong rather than either side being wrong, and correcting it gave the point to both.

Asking this server. One resolve_task_context call with the same sentence, and the whole response counted — summary, files, conventions, verification commands, every byte that would enter the context window. Run twice, once on each tier, against the same server binary.

Counting. Characters of real output divided by 4. Searching runs in-process rather than shelling out to rg, so the numbers reproduce on a machine that does not have it. A task counts as correct when the file that genuinely owns the change is among the files returned.

Every task, both sides

The files each side came back with are listed as returned. The interesting rows are the ones where the phrase grep costs nothing and the word-by-word fallback costs everything.

Free and Pro

Every file in a project is indexed at every tier, under its name and its path. A file the index has never heard of cannot be returned at any price, so that is not what the paid tiers sell — and an earlier version of this server did sell it by accident, stopping its walk at the free ceiling part-way through the first directory. On dub that left a whole directory of components invisible.

What a tier raises is how many files are read inside. The free tier reads 500 of them, chosen across the whole tree rather than the first ones found, and every answer says so. On dub that is worth exactly one task in 8: free finds 6, Pro 7. The one that turns is the copy-button task, where the owning component was in the index under its name but had not been read inside — so Pro answers from what the file says, and free from what it is called.

Pro also answers in 237 tokens against 293, because it has less to hedge about. Even so, the reason to pay is impact analysis, the dependency dashboard and duplicate detection. One task in eight is a real difference, not a paywall on being found.

What it comes to

At 12 agent tasks a day over twenty-one working days, discovery alone costs $20.63 a month against $0.37 on the free tier.

Those dollars are the token counts priced at Claude Opus 5 input rates ($5.00 per million) on the day the benchmark ran. Model prices move and the model you use may not be that one — the tokens are the measurement, the dollars are one way of reading it.

Prompt caching does not rescue the difference. Caching helps with text that repeats across prompts — the instructions file is exactly that, and genuinely gets cheaper after the first hit. Search output is the opposite: different query, different results, every prompt uncacheable by construction. The 16,372 tokens are paid at full price each time.

What this does not show

• One codebase, eight tasks. dub is one project with one set of naming habits. The harness ships with the package so you can find out what yours does.
• On the free tier, accuracy is a tie. Searching and the free tier each find 6 of 8, on partly different tasks. Pro finds 7. The headline claim on this page is cost, not accuracy.
• Not a transcript. This measures the context required to answer the question, not what one agent happened to spend on one afternoon. A real run varies with the model, the prompt and the mood of the sampler — that would make a better headline and a worse benchmark.
• Discovery only. The tokens spent writing the change are the same either way. Nothing here claims otherwise.
• Output tokens are excluded. Only what enters the context window is counted, at input pricing.

Run it on your own project

Start by reproducing this page. The harness and the exact task list are both in the repository, so the numbers above should come back unchanged on your machine.

git clone --depth 1 https://github.com/dubinc/dub
npm run benchmark ./dub/apps/web benchmark/tasks.dub.json

Then bring your own prompts. The eight above describe dub, and running them against a different codebase would measure how closely your project matches someone else’s vocabulary — not a question anyone asked. Eight prompts of your own is plenty.

npm run benchmark /path/to/your/next-app ./my-tasks.json

[
  {
    "id": "checkout-total",
    "prompt": "Fix the bug where the checkout total ignores the discount code",
    "searchTerms": ["checkout total", "discount code"],
    "expectPathContains": ["checkout"]
  }
]

prompt is what you would actually type — a symptom, not a filename. searchTerms is the phrase you would grep first, and it is the baseline’s best shot, so give it a real one. expectPathContains is the module that owns the task; a run counts as correct when a returned path contains it.

Write the prompts before you know how either side does. Choosing them afterwards turns a benchmark into a demo. Nothing leaves your machine.

The run is deterministic — no model call, no network, no sampling. Two runs on an unchanged checkout produce a byte-identical file, which is what makes the figures on this page checkable rather than merely quotable.