Skip to content

Evaluate native MCP routing instruction profiles#320

Draft
hmishra2250 wants to merge 1 commit into
mainfrom
hm/router-card-mcp-instructions-experiment
Draft

Evaluate native MCP routing instruction profiles#320
hmishra2250 wants to merge 1 commit into
mainfrom
hm/router-card-mcp-instructions-experiment

Conversation

@hmishra2250

@hmishra2250 hmishra2250 commented Jul 18, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • preserve stock MCP instructions as the default
  • add authenticated and keyless capability-honest experimental profiles
  • deliver profiles only through native MCP initialize.instructions

Experiment result — do not promote to default

P3 ran 384 immutable Convex-backed traces across Claude Sonnet 5 and Codex Terra using naturally discovered stdio MCP configuration with no Firecrawl CLI, skills, context card, forced --mcp-config, or injected instruction file.

  • authenticated router profile: 0% → 0%, 0pp on both harnesses, adjusted p = 1.0
  • keyless router profile: 0% → 0%, 0pp on both harnesses, adjusted p = 1.0
  • safety: 1/96 matched regression; simultaneous upper bound 6.14%, above the 5% gate
  • a separate eight-cell mechanism diagnostic found no behaviorally accessible profile-specific startup text in either harness

Recommendation: reject initialize.instructions as a router-card delivery surface for natural Claude Code and Codex CLI sessions. Keep stock instructions capability-honest, but do not merge these profiles as a routing feature unless maintainers explicitly want research-only profile support.

Decision artifact: Convex kg23nxb3s0gvqg8pcdqc2szqwh8arz95. Diagnostic artifact: Convex kg27gt5j74h2d59jzzcp7s5zx98arfqp. Raw traces remain in Convex.

Verification

  • npm test (11 passed)
  • npm run lint
  • GitHub build check passes

@hmishra2250 hmishra2250 changed the title Add native MCP routing instruction profiles Evaluate native MCP routing instruction profiles Jul 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant