Hi! I maintain mcpgrade, an open-source scorecard
for MCP server agent-usability, and recently benchmarked 36 popular servers
(results). I re-verified against
firecrawl-mcp 3.22.4 today before filing this.
The headline: 132 of your parameters ship with no description — url, formats,
jsonOptions, queryOptions on firecrawl_scrape, and so on across the catalog. The model
gets a name and a type, nothing else. This pattern usually comes from zod schemas where
.describe() was never added, so the fix is mechanical rather than architectural.
Two smaller things the scan flagged:
firecrawl_monitor_checks vs firecrawl_monitor_check differ by one character — models
mix near-twin names up.- 49 complex parameters have no example value in their description.
Why I think it's worth an hour: when I ran a live-model eval against an earlier version of
this catalog, tool-selection misses landed exactly on near-twin names, and on out-of-scope
tasks the model wrongly "found" a tool to call ~50% of the time — with large catalogs,
under-documented tools make agents do something when they should do nothing.
If you add descriptions and want the public table updated, comment here or open a rescan
issue on my repo and I'll happily re-run and publish the new score. And if you think any
rule is wrong, I'd genuinely like the pushback — the ruleset is opinionated by design.
Hi! I maintain mcpgrade, an open-source scorecard
for MCP server agent-usability, and recently benchmarked 36 popular servers
(results). I re-verified against
firecrawl-mcp 3.22.4 today before filing this.
The headline: 132 of your parameters ship with no description —
url,formats,jsonOptions,queryOptionsonfirecrawl_scrape, and so on across the catalog. The modelgets a name and a type, nothing else. This pattern usually comes from zod schemas where
.describe()was never added, so the fix is mechanical rather than architectural.Two smaller things the scan flagged:
firecrawl_monitor_checksvsfirecrawl_monitor_checkdiffer by one character — modelsmix near-twin names up.
Why I think it's worth an hour: when I ran a live-model eval against an earlier version of
this catalog, tool-selection misses landed exactly on near-twin names, and on out-of-scope
tasks the model wrongly "found" a tool to call ~50% of the time — with large catalogs,
under-documented tools make agents do something when they should do nothing.
If you add descriptions and want the public table updated, comment here or open a
rescanissue on my repo and I'll happily re-run and publish the new score. And if you think any
rule is wrong, I'd genuinely like the pushback — the ruleset is opinionated by design.