Declared is not implemented

Every MCP client announces what it can do. The initialize handshake carries a capabilities object: sampling, elicitation, roots, and so on. It is the obvious place to look if you are writing a server and want to know what your host supports.

It is not sufficient, and this week we found out how far it falls short — including in our own code, which was announcing several things it did not do.

Two clients, and they have almost nothing in common

We built a probe that measures a client by being a server it connects to. It records the handshake, then sends notifications/tools/list_changed and watches whether the client re-reads the tool list.

That notification matters more than it sounds. It is how a server tells a host "my tools changed" — and it is the entire basis of dynamic tool loading. A client that ignores it will connect to a server successfully and then never see the tools appear.

Cursor honours it. Zero tools/list calls in 38 seconds of silence, then one 10 milliseconds after the notification. That is unambiguous.

Devin declares nothing whatsoever — an empty capabilities object. That is legal, and it means a server has no way to know from the handshake whether elicitation or sampling will work. You find out by trying.

We could not establish Devin's list_changed behaviour. It was sitting in Restricted Mode with eleven extensions disabled, and we are not willing to report "ignores the notification" when the surface that would have responded was switched off. That is a result about our test conditions, not about the client.

Neither client declares sampling. If you are writing a server that asks the host to run a completion on its behalf, it will not work in either of these.

The same gap, in our own API

While building the probes we kept getting confident, wrong answers out of our own registry API.

?q=postgres returned results. They were not filtered — the parameter is search, and Fastify silently drops query params a route schema does not declare. Same with ?sort=downloads, which is sortBy. A 200 response with sensible-looking JSON is not evidence that the server understood the request.

That cost us a corpus. A measurement of "the 300 most-downloaded servers" was nothing of the kind, and the number derived from it went into a README before anyone noticed.

And in our own site

The server pages render tool names and descriptions collected by connecting to third-party servers. They were interpolated into HTML without escaping.

This was not hypothetical. chrome-devtools-mcp describes a tool as "…or select an option from a <select> element", and mcprating.io was emitting that as a real <select> tag — one raw occurrence in the live page, zero escaped. A description containing <script> would have run on our domain.

We publish a scanner for poisoned tool descriptions. We were rendering those same descriptions unescaped.

Two more, found the same week:

• Our rate limiter returned HTTP 500 instead of 429. The plugin throws whatever its error builder returns; ours returned a plain object with no statusCode, so it fell through to the generic error handler. Clients could not tell "slow down" from "this server is broken", and had no Retry-After to back off with.
• Our cache TTLs were ten minutes on data that changes weekly. The database scales to zero after five minutes idle, so the cache expired, a visitor woke the database, and compute was billed for another five minutes to return numbers identical to yesterday's. It never slept. That is what exhausted the free tier — not traffic.

The instruments were wrong more often than the systems

This is the part worth generalising from. Across two probes we found eight bugs in our own measurement code, and every one of them produced a confident result that was false in the flattering direction.

The two worst:

A 3B model scored as a success. The verdict checked whether the model had called mcp_connect. It had — and then emitted 8,620 tool calls in a single reply, looping a three-call cycle 2,872 times without ever terminating. Reaching a step is not completing a task.

A client scored as honouring a notification it never saw. The probe fired on a four-second timer. Cursor's initial tool read landed four milliseconds later, and "a list after the notification" counted it as a response. No client answers in four milliseconds. The fix was to wait until the initial read had been observed before poking, making the separation structural rather than lucky.

The others, briefly: a failed connection reported as connected, because the check asked whether connect was called rather than whether anything was live; a turn limit that truncated a working run and blamed the model; a tool list fetched once at startup, so proxied tools never reached the model and every run necessarily stopped at "connected" — which we had already written up as a finding about models; client identity read before the handshake completed, so the report said no client had connected while one was demonstrably connected and answering; a notification that deadlocked waiting for a list a lazy client never sends; and a gateway message we changed, after which the next run succeeded — we nearly credited the change, and an A/B with the old wording showed the model terminated either way.

Eight. Each caught only by disbelieving the instrument rather than the system under test.

Our own data contradicts our own video

One more, because it is the sharpest illustration.

mcprating.io marks zhafron-mcp-web-search as "Did not start unattended". We connected to it twice while writing this post: 6.8 seconds, then 9.8 three days later — two tools each time. It also appears in one of our own demo videos, connecting in three.

Nothing is broken, and that spread is the point. The harness allows 45 seconds and supplies no credentials, so a cold npx resolution can eat the budget on a bad day and finish comfortably on a good one. The stored verdict rests on one attempt, recorded on 16 September — and one attempt is thin evidence to publish across every page in the index.

We now show verification on every server page with the attempt count and the method attached, precisely so a reader can weigh it. "0 of 1 attempt succeeded" is a much weaker claim than a red badge, and it should read that way.

What we don't claim

• Not a compatibility matrix. Two clients, one of them inconclusive. It is the start of one.
• Not that Devin ignores tools/list_changed. We could not test it properly. Restricted Mode is our test condition, not their behaviour.
• Not that declaring nothing is wrong. An empty capabilities object is legal. The point is only that it tells a server author nothing.
• Not that these bugs are unusual. Every one is ordinary. The reason we found them is that we were measuring, and measurement makes ordinary bugs visible.
• Not that our instruments are correct now. They are correct against the cases we thought to check. The eight above were all things we had not thought to check.
• We are not neutral. We run this registry and the gateway these probes test, and several of the failures above are ours.

Reproduce it

The client probe is in the gateway repo. Point any client at it:

{ "mcpServers": { "probe": {
    "command": "node",
    "args": ["/abs/path/demo/client-probe.mjs"] } } }

Start the client, then read demo/client-probe-report.json. It records the handshake, counts tools/list before and after the notification, and says which path it took.

If you run it against a client we have not tested, we would like to see the report.

The gap between what a system announces and what it does is not a bug in MCP. It is the normal condition of software, and the only defence is to check.