Local models are ready for MCP. The servers aren't.

There is a plausible story about MCP and local models that goes like this: the protocol works, the models are getting good at tool use, so the only thing left is time. We ran it and found the bottleneck is somewhere else entirely.

Three models, one task — find an MCP server that can search the web, then connect to it — against a gateway fronting our index of 63,168 servers. The biggest model drove the protocol flawlessly and still failed the task, because every server it picked was broken.

What we actually ran

A script that is deliberately boring: it spawns the gateway over stdio, hands the model its 13 meta-tools as function definitions, and executes whatever the model calls. That is what a bridge like ollmcp does internally, reduced to something scriptable so the answer is a transcript rather than an impression.

Ollama has no MCP support of its own — the request has been open since November 2024 — so anything using MCP with a local model goes through a bridge. Ours is that bridge, minus the interface.

3B: it never stops

llama3.2 returned 8,620 tool calls in a single response. The first two were correct — mcp_discover, then mcp_connect. Then it locked into a three-call cycle, list_active → call_tool → disconnect, and repeated it 2,872 times with malformed arguments, never emitting a final answer.

Generating that took about nine hours on CPU.

This is a termination failure, not a comprehension failure, and the distinction matters: the model understood the tools well enough to use the right two in the right order. It could not tell when to stop.

A real client would have caught this — ollmcp gates tool calls behind human-in-the-loop confirmation, so a person would have stopped it at the first call. They would have lost thirty seconds instead of a night, and still not got an answer.

8B: the protocol is not the hard part

gemma4 at 8B was genuinely competent. Over five turns it:

• searched, got 2,238 results, picked one
• hit "No install command available" and tried a different server
• hit a [community] trust warning and worked out on its own that the fix was to re-issue the call with confirmed: true
• hit MCP error -32000: Connection closed, stopped, and explained accurately what it had tried

Every move was right. It ended with nothing connected, because the server it chose does not start. Given a task it could finish, it finishes — see below.

31B: four servers, four failures

The 31B ran the same task in 13 turns at roughly a third of a second each, and we ran it five times with identical results. Here is the whole trajectory:

 1  discover "web search"          2,238 results
 2  connect  duckduckgo-web-search → trust warning
 3  connect  …confirmed            → Connection closed
 4  discover "google search"
 5  discover "brave search"
 6  connect  brave-search-mcp      → trust warning
 7  connect  …confirmed            → Connection closed
 8  connect  pipeworx-brave-search → No install command
 9  discover "tavily search"
10  connect  mcp-tavily            → trust warning
11  connect  …confirmed            → Connection closed
12  discover "search"              5,489 results
13  connect  npx @modelcontextprotocol/server-everything → connected

Four different registry servers. Four failures. After the fourth it stopped searching our index and connected to a package it already knew from training, by passing an explicit command instead of a slug.

That last move is the finding. Offered thousands of candidates, the most capable model we tested concluded the index was not worth another query and fell back to something it remembered. It was right to.

We measured a 36% connect rate earlier this month and argued the failures were mostly our harness being hostile. This is the same number arriving from the other direction, with no harness to blame: a model choosing freely, by quality score, hitting four dead servers in a row.

Change the question, and it works in four turns

Everything above asked for a capability — a server that searches the web. Ask instead for any server it can connect to, and the same model completes the entire loop:

1  discover "everything"
2  connect  everything-slim              → trust warning
3  connect  …confirmed                   → connected · 6 new tools appear
4  everything-slim__echo("Hello MCP!")   → "Echo: Hello MCP!"
   DONE

Four turns, three runs out of three, identical. That fourth call is the part worth pausing on: everything-slim__echo did not exist when the conversation started. It appeared only because the model connected to the server, and the model then called it by name. Discovery, connection, and use of a tool it could not have known about — which is the entire design, demonstrated by a model running on someone's own hardware.

The 8B does it too, on a CPU, in five turns. It took the other available route — the mcp_call_tool meta-tool rather than calling the proxied tool directly — got the arguments wrong on its first attempt, read the validation error, and fixed them:

4  mcp_call_tool(utility-mcp-tools__notify, {})   → "message" is Required
5  mcp_call_tool(utility-mcp-tools__notify, {message: "…"})  → {"notified": true}
   DONE

So the working range starts lower than the hardware bill suggests. An 8B model is a laptop, not a cluster. Somewhere between 3B and 8B this stops being a research question and starts being a setup instruction.

Two things keep this honest. everything-slim is a test server — echo, add, toggle — so what is proven is the mechanism, not that it did useful work. And the contrast with the section above is the whole point: the machinery works on the first attempt; the catalogue of real servers does not.

We also nearly failed to discover this at all, for a reason covered below.

A useful side effect: the token estimate holds

Every context-cost figure we publish is sized as chars / 4, which is an estimate. Inference servers report a real tokenizer count, so this run checked it:

Within 1–2% on both gemma tokenizers and 12% high on llama. The estimate never came in under the real count, so figures derived from it overstate cost rather than flatter it.

Our own harness was wrong five times

Every one of these would have produced a confident, false published claim, and we only caught them by reading transcripts instead of summaries:

• it scored the 3B's 8,620-call meltdown as success, because the check was "did it call mcp_connect" rather than "did it finish"
• it reported connected: yes for a run whose connection returned Connection closed — same bug, different shape. It now asks the gateway what is actually live
• a turn ceiling of 8 truncated a run that was still making progress, and the verdict blamed the model for our constant
• we changed a confusing gateway message, saw the next run succeed, and nearly wrote that the change fixed it. An A/B with the old wording restored showed the model terminates either way. The change stands on design grounds; the loop that prompted it remains unexplained
• and the worst one: the probe read the tool list once, at startup. The gateway's whole mechanic is that a server's tools appear only after you connect. So the proxied tools never reached the model, every run necessarily stopped at "connected", and we had already written that up as a limit of the models. It was a limit of our script. Real clients re-read on tools/list_changed; ours does now, and the result reversed

What we don't claim

• Not a benchmark. One task, one prompt, three models. It shows what happened, not what happens.
• Not that 3B models can't use MCP. One 3B model failed one task. The termination failure may be specific to llama3.2, to this prompt, or to having 13 tools in front of it.
• Not a speed comparison. The 3B and 8B ran on CPU and the 31B on two H100s. The ~100× gap is hardware, not capability, and says nothing about either model.
• Not that those four servers are broken for everyone. We supply no credentials and allow no interactive setup. Some of them likely work fine with an API key their author expects you to have.
• Not that the full loop works for real work. Both the 8B and the 31B completed discover → connect → call, but against everything-slim and utility-mcp-tools — servers whose tools are echo and notify. The mechanism is shown. Doing something useful with it is not.
• Not that 8B is the threshold. One 8B model passed and one 3B model failed. The line is somewhere between them and we tested two points, not a range.
• Not that the 31B would have found a working search server eventually. We stopped at 20 turns. It might have; it had already stopped looking.
• Not a tokenizer result for Claude. The calibration is llama's and gemma's. It supports the ballpark, not an exact transfer.
• We are not neutral. We run the registry the models were searching and the gateway they were driving. The script is in the repo and the transcripts are reproducible; argue with the method.

Reproduce it

npx -y @mcp-rating/gateway   # the 13 meta-tools under test

The probe is in the gateway repo. Point it at Ollama, or at any OpenAI-compatible server:

OLLAMA_HOST=http://your-host:8000/v1 node demo/ollama-probe.mjs your-model

One warning from experience: if you are using vLLM and see zero tool calls and a chatty prose answer, that is the server, not the model. Tool calling needs --enable-auto-tool-choice and a --tool-call-parser. We nearly recorded a serving misconfiguration as a model failure.

The models are further along than the servers they are being pointed at. That gap is the whole story, and it is not the models' fault.