Appearance

System follows your phone’s setting. Default.

Custom MCP
Date
Read
21 min
Views

Too many MCP tools? I made the agent search instead, and measured 334 sessions. Here's what it fixed and what it didn't.

Oct 2, 2026 · 21 min read ·

334 Claude Code sessions on a real HubSpot. Skill and tool selection as MCP tools, picked by a decision model: right 24 of 24, $0.0005 a route, 12x less context, paid once per session.

01 · The symptom
“Every session loaded 88 tool definitions to use five of them. When the model wanted a skill, it guessed the name.”

I started with a proof of concept: a decision model picking the right sales skill from inside an MCP server, on a mock CRM. This is the real one. I built tool selection and skill selection as MCP tools on my production HubSpot MCP server, put TypeSafe’s Jev in charge of the picking, and ran 334 headless Claude Code sessions against my own portal.

It’s the same idea as Anthropic’s Tool Search Tool, done inside the MCP server. The host loads four tool definitions instead of 88 and the model searches for the rest. People call it progressive disclosure. I wanted to know what it’s actually worth.

Five questions. Is the pick right? Does it come with a probability I can act on? What does it cost? How long does it take? And does it stop the host from loading every tool and every skill at the start of every session?

Four came back clean. The fifth is the honest part of this post. The host loaded 12 times less context, and a single task still cost more, because prompt caching had already made the big tool list cheap. Then a sixth question, asked after the first write-up, softened that: in a conversation with a second and third turn, the routing is paid once and reused. The arithmetic for both is below.

02 · The gap

What I set out to prove

An MCP server hands the host every tool it has. Mine has 88, plus HubSpot’s nine sales skills. The host loads the list once per session, and the model reads it on every turn. A rep asking for a morning brief needs one skill and five tools. The session gives the model all of it and lets it pick.

The alternative is to make selection itself a tool. The server exposes a handful of meta-tools. The model says what it wants in the user’s own words. A cheap, fast decision model returns the skill, the charter and the tools that fit, each with a probability. The model loads what it needs and gets on with the job.

That claim has four parts, and I wanted each one measured on real sessions, not on a diff of the tool list:

  • Accurate. The first pick is right nearly always, and when it isn’t, the right answer is still in the list.
  • Scored. A probability per option, so the server can set a threshold, the model can see the runner-up, and I can audit the misses afterwards.
  • Cheap and fast. A rounding error on the session, in money and in seconds.
  • Less context. Four definitions loaded instead of 88 and nine skills’ worth of text, and the model searches for the rest.

The proof of concept measured the tool list and the routing accuracy on labelled prompts. It never measured a session. This time I did.

03 · The fix
Custom MCP

Four tools instead of 88

The server now serves two surfaces from one process. The original endpoint doesn’t change: 88 tools and the write gate. The new one, /mcp/routed, advertises four tools.

find_capabilities takes the request in the user’s words. It returns the matching sales skill, the matching specialist charter and the HubSpot tools they need, each with a probability. The tools come with their input schemas inline, and a one-line next_step tells the model what to load.

load_skill returns one of HubSpot’s nine skills, byte for byte, behind a short header that maps the vendor’s connector tool names onto mine and explains the write gate. load_charter returns one of the 44 specialist charters the server already shipped as MCP prompts, which hosts never pull on their own.

call_hubspot runs any of the 88 tools by name. It checks the arguments against the tool’s JSON schema first, then dispatches into the exact function the registered tool uses. Previews, approval tiers, undo and the audit log are identical on both surfaces.

YOU CLAUDE MCP SERVER JEV WRITE GATE Change the amount on one deal to 12,500 find_capabilities hands over the ask 134 questions 2 choices, 132 yes/no Odds per option objects 0.90 Skill, charter, tools odds + input schemas load_charter, then call_hubspot(update) Preview, CONFIRM amount is sensitive You approve or not nothing changed yet Same handlers, same gate: call_hubspot dispatches into the code path the 88 registered tools use. What changed is what the host loads first: 4 tool definitions instead of 88, and a skill or charter the model reads.
01 · You Change the amount · on one deal to 12,500 02 · Claude find_capabilities · hands over the ask 03 · MCP server 134 questions · 2 choices, 132 yes/no 04 · Jev Odds per option · objects 0.90 05 · MCP server Skill, charter, tools · odds + input schemas 06 · Claude load_charter, then · call_hubspot(update) 07 · Write gate Preview, CONFIRM · amount is sensitive 08 · You You approve or not · nothing changed yet Same handlers, same write gate. 4 tool definitions loaded instead of 88.
One routed request. Claude hands the ask to find_capabilities, Jev answers 134 typed questions in one call, and the matches come back with their schemas. Claude loads the skill and writes through call_hubspot, into the same gate as the other 88 tools.

The router is Jev, TypeSafe’s evaluation-only model, reached through Vercel’s AI Gateway. One HTTPS request carries 134 questions: pick one of the nine skills or none, pick one of the 44 charters or none, and a yes or no for every skill, charter and domain tool. Jev returns a probability for each. Anything at 0.5 or above comes back, plus the chosen skill’s and charter’s tools.

If the gateway fails or the key is missing, the server falls back to the keyword router it already had and says so in the response. That fallback turned out to matter.

The same selection sits on the full surface too, as two ordinary tools: hubspot_find_skills and hubspot_load_skill. A host that loads all 88 tools still gets routed skills. That let me test the skill idea with and without the small tool list.

04 · The method

334 sessions against my own portal

Every number here comes from headless Claude Code sessions against the PromptMetrics HubSpot portal. Same isolation flags, same operator framing on the prompt, arms interleaved so time of day and HubSpot latency hit them equally.

A runner parsed each transcript for exact per-call token usage, the tool-call sequence with timing, turns, duration and cost. It checked completion against the portal, then rejected pending previews and undid every auto-tier write so the next session started clean. Writes only ever touched one test contact and one test deal.

RunQuestionSessions
Surface A/BFull surface, routed surface, no MCP. 12 tasks, three times each106
Skill benchFour skill-driven tasks on both surfaces24
Router comparisonJev against keywords on the routed surface. 20 tasks, three times each120
Sonnet follow-upThe four skill tasks, Jev against keywords, on Sonnet 524
ReuseTwo three-turn conversations on both surfaces, plus each turn on its own60

Two hung control prompts in the A/B were dropped. Everything but the Sonnet run used Claude Opus 5.5, the Claude Code default that day. Costs are the API-equivalent at Anthropic’s list prices, as Claude Code reports them per session. The sessions themselves ran on a subscription. Both arms of every comparison used the same model, so repricing moves the totals, not the comparison.

05 · Accuracy

Right every time, with a score next to it

Jev, skills, Opus 5.5 24 of 24 Keywords, skills, Opus 5.5 14 of 24 Jev, charters, Opus 5.5 35 of 42 Keywords, charters, Opus 5.5 32 of 42 Jev, skills, Sonnet 5 12 of 12 Keywords, skills, Sonnet 5 9 of 12
Jev, skills, Opus 5.5 24 of 24 Keywords, skills, Opus 5.5 14 of 24 Jev, charters, Opus 5.5 35 of 42 Keywords, charters, Opus 5.5 32 of 42 Jev, skills, Sonnet 5 12 of 12 Keywords, skills, Sonnet 5 9 of 12
Right pick on the first lookup of each live session. Jev: 24 of 24 on Opus, 12 of 12 on Sonnet. Keywords: 14 and 9. On charters the two are close, and six of Jev's seven misses were sessions where it chose a skill instead, which is what the design asks for.

Offline first, because live tasks have to be sparse. On 40 labelled prompts, 16 of them written to sit between two skills, Jev picked the right skill first 36 times, and 13 of the 16 boundary cases. Keywords managed 17 and 4. On the server’s 49-prompt charter corpus, written for the keyword router and full of literal requests like “list workflows”, the two were level: 46 and 47 of 49.

Then live. I ran two copies of the routed server, one on Jev and one with the keyword fallback forced on, and sent 20 tasks through each three times. On the 24 sessions whose task names a skill, Jev’s first pick matched the label 24 times. Keywords matched 14.

Every keyword miss had the same shape: a prompt with no trigger phrase in it. “I spoke to Chris this morning, legal is done on their side” never says log a call. “Which deals are closing this month and which have gone quiet” never says pipeline. Keywords returned no skill at all for those, and put call prep above log-call for a prompt that began “just got off a call with”. Jev got every one.

Jev, all 40 prompts 36 of 40 Jev, 16 boundary prompts 13 of 16 Keywords, all 40 prompts 17 of 40 Keywords, 16 boundary prompts 4 of 16
Jev, all 40 prompts 36 of 40 Jev, 16 boundary prompts 13 of 16 Keywords, all 40 prompts 17 of 40 Keywords, 16 boundary prompts 4 of 16
The offline eval before the live runs: 40 labelled prompts, 16 on the boundary between two skills. Jev 36 of 40 and 13 of 16. Keywords 17 and 4.

On charters Jev was right 35 of 42 times against 32, and the gap is smaller than it looks. Six of Jev’s seven misses were sessions where it chose a skill and no charter, because the prompt was a skill request and the label also allowed a charter. The keyword misses were generic beating specific: “open deals with no next activity” went to lists, and “list our active workflows” went to objects, with workflows second.

The probability is what makes a miss cheap. Here’s what the live server returns today for three requests, placeholder names only:

find_capabilities, live
"Just got off a call with Acme: they agreed the 12-month term, contract goes to their CFO next week. Log it."
  skills:   log-call 1.00, hubspot 0.70     charter: engagements     tools: 26     739 ms, $0.0005
  next_step: Call load_skill(name="log-call") and follow that workflow …

"Which deals are closing this month and which ones have gone quiet?"
  skills:   pipeline-pulse 1.00, hubspot 0.85     charter: objects     tools: 39     559 ms
  next_step: Call load_skill(name="pipeline-pulse") …

"What's the weather in Berlin?"
  skills:   none 1.00     charter: none     tools: 0     713 ms
  next_step: No HubSpot charter matches. … otherwise answer without HubSpot.

A wrong keyword pick usually cost a detour, not the task. The model read past the first result and took the right skill from second place on every log-call rep. Where the right skill wasn’t in the list at all, it failed all three reps of that task, because it worked from a charter without the skill’s “gone quiet” logic. The right answer being in the list, with a score next to it, is the difference between a slower session and a wrong one.

Sonnet 5 repeated the pattern on a smaller model: Jev 12 of 12, keywords 9 of 12, the same wrong pick on the log-call prompt. Sonnet recovered too, but paid more for it. With Jev its sessions ran two turns shorter, 14% cheaper and 16% faster at median, with no malformed tool calls against four.

On Opus the keyword arm had been slightly cheaper per session. Twelve sessions per arm is a direction, not a measurement, but it points the way you’d expect. The less capable the model, the more the first pick matters.

06 · Cost

What a route costs

Over 57 live routes on the Opus run, Jev answered every one. Median latency was 651 ms, p95 1,079 ms. Each route cost $0.0005, about 0.2% of the session it sat in. The Sonnet run’s 12 routes came in at 780 ms median and 1,114 ms p95, for the same $0.0005, or 0.3% of the session.

That’s 134 typed questions, answered in one call, for half a tenth of a cent. The routing bill for all 60 Jev sessions in the router comparison was $0.027. The keyword router is free, instant and sends nothing anywhere, which is why it stays as the fallback and as the opt-out for anyone who doesn’t want request text leaving the server. Nothing else leaves either. Jev sees the user’s request and the skill and tool descriptions, never portal data.

One lesson came free. My second attempt at the Sonnet run put 24 sessions through with both arms on keywords, because the Jev instance had been restarted in a shell without the gateway key, and the fallback is silent by design. The runner now probes each arm’s router at startup and refuses to run on a mismatch. A fallback that works too well needs a tripwire.

07 · The catch

Less context, bigger bill

No MCP 3,304 tokens Full surface, 80 tools 17,944 Routed surface, 3 tools 4,517
No MCP 3,304 tokens Full surface, 80 tools 17,944 Routed surface, 3 tools 4,517
Context in the first request, median over 106 sessions. Claude Code itself is 3,304 tokens. The full surface adds 14,640 on top. The routed surface adds 1,213.

The context claim held outright. Before the model does anything, the routed surface adds 1,213 tokens where the full surface adds 14,640. Across a whole session it carries a third less context per API call, 12,906 against 19,242, because the tool list rides along in every request and the routed one is tiny. The tools/list today is 33.7 KB for 88 tools and 2.4 KB for four.

Per task, mediansNo MCPFull, 80 toolsRouted, 3 tools
Context in the first request3,30417,9444,517
Context per API call3,30419,24212,906
API calls2712
HubSpot tool calls, excluding routing044
Cost$0.024$0.067$0.147
Duration8 s20 s29 s
Tasks completedn/a36 of 3636 of 36

Then the bill said the opposite. The routed surface cost 1.65 times more per task and took 1.4 times longer. Both surfaces finished every task, with identical write-gate behaviour, so this isn’t a capability gap. It’s prompt caching.

Full, 80 tools $4.29 Routed, 3 tools $7.07 cache reads cache writes
Full, 80 tools $4.29 Routed, 3 tools $7.07 cache reads cache writes
Context tokens over 36 sessions per arm, split into cache reads at $0.20 per million and cache writes at $8. The full surface read 6.37M from cache and wrote 0.78M. The routed surface read 5.19M and wrote 1.45M. Fewer tokens, higher bill.

How does context fall 12 times while cost rises? Because “context” above is the first request, and the bill is set by four things the first request doesn’t show.

The big tool list is cached, so it’s nearly free after the first turn. The host sends it with every request, but the API caches it and charges a cache read at a tenth of the price: $0.20 against $4 per million tokens on Opus. In the full arm, 89% of all context tokens were cache reads. The 14,640 tokens of definitions are written once per session and read cheaply after that. The tax I set out to remove was already mostly discounted.

What routing adds isn’t cached when it arrives. The lookup result, with its two dozen schemas and the skill or charter text, lands in the conversation as new context. New context is a cache write, and Claude Code’s one-hour cache writes cost $8 per million, twice list price. Routing removed cheap tokens and added expensive ones.

Then there are more turns. The lookup is a round trip, loading the charter is another, and on write tasks the charter told the model to verify what it wrote. Twelve API calls per task against seven, and every call re-sends the whole conversation. And more output. The routed arm produced 80% more output tokens, at $20 per million the dearest token there is.

Over 36 sessionsFull, 80 toolsRouted, 3 tools
Cache reads, $0.20 per million6.37M tokens5.19M
Cache writes, $8 per million0.78M1.45M
Total context tokens7.15M6.64M
Output tokens, $20 per million5,8517,269
Cost$4.29$7.07

The routed arm used 7% fewer tokens in total and cost 65% more. It moved two thirds of a million tokens from the cheap column to the expensive one and spent more on output. It saved tokens where they were already cheapest and spent them where they were dearest.

So the context claim is true and, for a single task on one server with one model and caching on, financially beside the point. Two things change that picture. I measured the first and collected the evidence for the second.

08 · Reuse

Load once, reuse all session

One task per session is the worst case for routing. The lookup result and the skill text are paid for once and used once. Real work isn’t one turn. A rep logs a call, then asks about the deal, then asks what’s still open.

So I ran 60 more sessions. Two three-turn conversations, log a call three times and look a contact up then ask two follow-ups, on both surfaces, plus every prompt as its own session for comparison. Each turn ran as its own Claude Code process resumed into the same conversation, so the token usage per turn is exact.

The model never asked twice. In 24 follow-up turns across 12 three-turn sessions, zero lookup calls and zero skill loads, on either surface. What turn 1 put in the conversation stayed there, and the model used it.

Routed, 4 tools Turn 1, 1 lookup, 1 load $0.182 Turn 2, no lookup $0.036 Turn 3, no lookup $0.043 Full, 88 tools Turn 1, lookup in half the sessions $0.118 Turn 2, no lookup $0.034 Turn 3, no lookup $0.041
Routed, 4 tools Turn 1, 1 lookup, 1 load $0.182 Turn 2, no lookup $0.036 Turn 3, no lookup $0.043 Full, 88 tools Turn 1, lookup in half the sessions $0.118 Turn 2, no lookup $0.034 Turn 3, no lookup $0.041
Cost per turn in three-turn sessions, medians over six sessions per surface. Turn 1 pays the lookup and the skill text. Turns 2 and 3 reuse them: no second lookup, no second load, about a fifth of the cost.

A follow-up turn cost about a fifth of the first: $0.036 to $0.043 against $0.118 to $0.182. It wrote a tenth of the cache, 2,412 to 4,327 tokens against 18,778 to 33,262, because the skill text and the tool list came back as cache reads. Five follow-up turns were answered correctly with no HubSpot call at all, from what the skill had fetched on turn 1.

Three requests in one session cost 30 to 65% less than the same three as separate sessions, and took a third less time. The saving was larger on the routed surface, 43 and 65%, than on the full one, 30 and 53%, because the routed surface pays more up front and pays it once.

Three requests, median costRouted, one sessionRouted, three sessionsFull, one sessionFull, three sessions
Log a call three times$0.381$0.674$0.310$0.441
Look up a contact, two follow-ups$0.140$0.403$0.107$0.230

The honest reading: the routed session still costs more than the full one, $0.381 against $0.310 on the write scenario. The gap shrinks with every turn. It doesn’t flip. Keep the work in one conversation and the routing becomes a fixed cost per session, not a tax on every request.

09 · Evidence

What a small tool list buys, and what it doesn’t

The question behind the whole experiment is whether a smaller context makes the model better, not just lighter. Three answers, kept apart: what I measured, what others measured, and what I expect but can’t show.

Measured here, on Opus 5.5. The model with 88 tools in view invented property names exactly as often as the model with four: 21 times against 24, all on one task, “which deal properties track why a deal was lost”, and only after listing the real properties first. Same behaviour, same count, both surfaces. Twelve easy tasks and one strong model, so it’s a null result, not proof that size never matters.

What did cost accuracy was the wrong option and the missing schema, not the token count. A wrong first pick from the keyword router added seven times the tool errors and 46% to the session’s cost on Opus, and two extra HubSpot calls per session on Sonnet 5.

In the run where a bug left the chosen skill’s tools out of the lookup result, the model guessed the arguments for a tool it had no schema for 80 times in 12 sessions. The proxy rejected every guess. The full surface, with the schema in context, produced none. Fewer wrong options and a schema for every tool in view: that is what moved accuracy in my transcripts.

Measured by others. Anthropic’s own MCP evaluation went from 49% to 74% on Opus 4, and from 79.5% to 88.1% on Opus 4.5, when tools were loaded on demand instead of all at once. Its docs say selection degrades past 30 to 50 tools. Chroma tested 18 models and found a focused 300-token prompt beat the same question inside a 113,000-token context on every one of them.

RAG-MCP found that retrieving the right tools beat loading all of them three to one, 43% against 14%, past a hundred tools. MCP-Bench found extra servers hurt small models and left frontier models flat. Every one of those studies changes two things at once, fewer tokens and fewer options, and none of them separates the two.

Expected, not proven. Headroom when several MCP servers share one host, where 88-tool lists compete for the window. Smaller models, where the Sonnet run already shows the first pick mattering more and the literature says length hurts them earlier. Long sessions, where definitions crowd out the work. And any host or model without prompt caching, where 12 times less context is 12 times less on the bill too.

None of my runs set those up. The design I measured is also the naive one: it returns 24 tool schemas where three would do and loads the charter in a separate turn. Both are a day’s work and a re-run.

10 · Budgets

Where vendor budgets bite

Every MCP call to a SaaS vendor spends something. I checked fifteen MCP servers and hosts. Four cap MCP calls directly, seven cap the API underneath, two cap the number of tools, and none publishes a per-conversation limit.

VendorWhat’s capped
PipedriveDaily API tokens per company, 30,000 times the plan multiplier times seats. A search costs 40, a list 20
HubSpotAPI requests, 100 to 190 per 10 seconds and 250,000 to 1 million a day
NotionMCP calls, 20 searches and 20 queries per 10 seconds
ZapierTwo tasks per tool call, from a monthly task allowance
SalesforceAbout 20 tools active at once, and the org’s 24-hour API allocation
Claude CodeTool search kicks in when deferred definitions reach 10% of the window. 25,000 output tokens per MCP result

An agent spends that budget eight to fifteen calls at a time. On Pipedrive a morning brief that fans out across lists and searches is expensive by design.

Routing helps in three places. The routing calls cost the vendor nothing: in the router comparison, 57 lookups and 60 skill loads produced zero HubSpot requests, and Jev’s bill for the run was $0.027. A right first pick avoids the detour, and every error or extra call after a wrong one is a vendor request the budget counts. And one function sees every vendor call, call_hubspot, so a per-portal counter, a soft limit with a warning in the response, or a queue when the burst limit is near is one change in one place. That counter isn’t built yet.

Where it doesn’t help: routing does not reduce the calls a task needs. The routed surface made the same median four HubSpot calls per task as the full one, and on two write tasks the charter’s verify-after-write rule raised it to seven. The saving is in wrong calls, not in calls. And a vendor’s own hosted MCP, Pipedrive’s or HubSpot’s, counts on their side, where no proxy can meter it. A self-hosted server talks to the REST API, so the API’s limits apply and the proxy can watch them.

11 · Behaviour

None of this works unless the model actually searches instead of guessing. Four things the transcripts showed.

It looks up first, once you tell it to. On the full surface, before the server instructions said to call hubspot_find_skills first, the model guessed a skill name, daily brief with a space, failed, then tried daily-brief. One instruction line fixed that in every session after. On the routed surface it called find_capabilities first in every session that had anything to do with HubSpot, and skipped it for the weather.

It reads the whole list, not the top line. Every time the keyword router put call prep first and log-call second, the model loaded log-call. The probability and the runner-up aren’t decoration.

It accepts correction. When the model sent an association lookup with a field missing, the schema check returned the exact missing field, and the model fixed its arguments and carried on. The full surface’s registered tools accept unknown arguments silently. The keyword arm, which started from the wrong charter more often, drew four of these corrections. The Jev arm drew none.

It follows the skill it loads. Call prep saved a prep note. Log-call recorded the call as a note, created the follow-up task and set the lead status, as HubSpot’s workflow says. Sonnet 5 went one further than Opus and set the lifecycle stage too. Every write went through the gate and the runner undid it. On the full surface, with all 88 tools in view, the loaded skill changed what the model did in exactly the same way.

That last one I didn’t expect. Serving a skill through a tool result works as well on a host that loaded everything as on one that loaded four tools. The small tool list and the served skills are separable. You can take the second without the first.

12 · The idea

Skills behind a login

The install problem is gone either way. Nobody copies a folder, and when HubSpot ships version 2.4 of its skills, the server updates them for everyone at once. Once a skill is something a server serves rather than a file a rep installs, a vendor has the three things it needs to charge for it.

Entitlement. The hosted surface authenticates every request and ties it to a portal. find_capabilities could return only the skills that account pays for, and load_skill could refuse the rest, with the same scored lookup for everyone.

Versioning. A skill is a current version, not a copy. Fix a workflow on Tuesday and every subscriber runs the fix on Tuesday. The mapping header that turns the vendor’s tool names into mine is the same idea applied to compatibility.

Metering. Every routed session leaves a trace of which skill was picked for which request, with its probability, and every vendor call passes through one function. That’s usage data a folder of markdown never produces, and it’s the same counter the budget section needs. Which skills earn their place, and who is spending what, from one place.

Put together, that’s a skill pack sold as a connector URL. A team adds the server, signs in, and the skills they pay for show up in the lookup. None of it is built. There’s no entitlement store, no billing, and the routed surface identifies a portal, not yet a person.

Two cautions before anyone runs with it. The skill text leaves the server in the tool result, so a subscriber can copy it. What the subscription sells is the current version, the routing, the live mapping onto the tools and the write gate, not a secret. And HubSpot’s own skills are Apache 2.0, so this applies to skills you write, not to theirs.

13 · Open

What isn’t proven yet

  • One model for the surface comparison. Opus 5.5 only. The Sonnet 5 run covered the router question on four tasks, not the surface question, so whether a smaller model benefits from the small tool list itself is still open.
  • One server on the host. The headroom benefit depends on several MCP servers sharing a session, and this bench didn’t set that up.
  • The hallucination result is a null result. No difference at 88 against 4 tools, on twelve easy tasks and one strong model. It doesn’t show that context size never matters.
  • Headless only. Claude Code declares inline confirmation, so confirm-tier writes came back as a request the headless client declined, and the gate recorded a reject. Interactive sessions would show the form and wait. Same code path on both surfaces, but I haven’t sat through it on the routed one.
  • An unoptimised design. Fold the charter into the routing result, return the top three schemas instead of 24, trim the verification boilerplate. Each one is a re-run.
  • Small live samples. 24 skill-labelled sessions per arm on Opus, 12 on Sonnet, 12 multi-turn sessions per arm. The offline eval is 40 prompts. Enough to see the shape, not enough to quote a percentage to two digits.
  • Local runs. The routed surface and the served skills are deployed, and the hosted routed endpoint has its OAuth resource registered, but every measurement here ran against a local instance of the same code.
14 · Lessons

What I’d tell anyone building this

  • Make selection a tool, and give it a probability. The score is what lets the model recover from a wrong first pick and lets you audit the misses afterwards. A router that returns one name is a router you can’t debug. And tell the model to look up first, or it guesses. One sentence in the server instructions does it.
  • Pay for the router. It’s the cheapest part. Half a tenth of a cent and under a second, for the difference between 24 of 24 and 14 of 24.
  • Keep a free fallback, with a tripwire on it. Keyword routing is the right opt-out for anyone who wants nothing to leave the server. It’s also silent when it takes over by accident. Probe for it.
  • Price the tokens you save at what they actually cost. With prompt caching, a static tool list is nearly free after the first turn. Count cache reads and cache writes separately before you claim a saving.
  • Keep the work in one session. The lookup and the skill are paid once and reused on every later turn. Three requests in one conversation cost up to 65% less than three sessions.
  • A tool without its schema is a guess. 80 rejected argument guesses in 12 sessions when the schema was missing, none when it was there. Return the schema with the pick, every time.
  • Return less than you can. Two dozen schemas inline were most of the context the routed arm spent. Return the skill and the top three tools, and let the model ask for more.
  • Serve skills as a tool result, not a prompt. Hosts don’t pull MCP prompts. A tool that returns the text, behind a header that maps the vendor’s tool names onto yours, works on every host, with a small tool list or without one.
  • Keep the gate shared. The only reason I can say both surfaces behave the same on writes is that they run the same function.

The code is on the main branch of hubspot-mcp: the routed surface, the served skills, the bench, the evals and the reports. The design and every earlier number are in docs/routed-mode.md, the reuse run in its report, and the evidence table with every source behind the benefits section in the content handoff. Run the bench against your own portal, never your live one, and if your numbers point the other way, I want to see them. Tell me where I’m wrong.

15 · What it saved
24 of 24
right skill on the first lookup, against 14 of 24 for keyword matching
$0.0005
per routing decision, in 651 ms, paid once per session and reused on every later turn
Reactions
Subscribe

Get the next MCP-gap post in your inbox. Every two weeks, no fluff.

More on the Custom MCP rung
Comments

Comments

Loading comments…

    Leave a comment

    Not shown publicly.
    Work with