Skip to content

Capture MCP tool calls in PostHog - #128

Open
masnwilliams wants to merge 3 commits into
mainfrom
hypeship/posthog-mcp-analytics
Open

Capture MCP tool calls in PostHog#128
masnwilliams wants to merge 3 commits into
mainfrom
hypeship/posthog-mcp-analytics

Conversation

@masnwilliams

@masnwilliams masnwilliams commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

Summary

Instruments the MCP server with PostHog MCP analytics (@posthog/mcp, pinned to 0.10.1) so every inbound tools/call, tools/list, and initialize is captured as a $mcp_* event — tool name, arguments, latency, error state, and client name/version.

What that buys us: per-tool call volume, error rate and latency percentiles, which clients connect, and which advertised tools never get called. Today we ship tool changes blind — kernel API traffic tagged X-Source: mcp-server shows endpoints, not tools.

  • src/lib/mcp/analytics.ts — module-scope posthog-node client plus instrument() wiring. identify attributes events to the Clerk user id when the request carries a JWT; API-key requests stay session-scoped rather than getting a made-up identity.
  • src/app/[transport]/route.ts — instrument inside the createMcpHandler callback, and drain the queue with after() so the flush runs once the response is already on the wire. Capture adds no latency to a tool call.
  • .env.examplePOSTHOG_PROJECT_TOKEN and POSTHOG_HOST. With no token set the whole thing is a no-op, so deploys and local dev without PostHog credentials are unaffected (dev logs a one-time error so the miss isn't silent).

POSTHOG_PROJECT_TOKEN and POSTHOG_HOST need to be set per environment before any events show up.

No payloads leave the server

Every tool serializes its result through jsonResponse, so a response reaches the SDK as a single JSON string and its key-name redaction can't see inside it — a newly created API key, a TOTP code, or a CDP URL would be captured verbatim. beforeSend therefore:

  • deletes $mcp_response on every event. Call metadata is what the analytics are for; response bodies aren't worth that risk on a public server.
  • deletes $mcp_parameters for manage_api_keys, manage_credentials, and manage_proxies, whose arguments carry arbitrary field/value maps (values: { username, pin, ... }) that key-name matching can't cover either.

Deliberately left off

instrument() can inject a required context argument into every tool schema to capture what the agent was trying to do ($mcp_intent). It's the most interesting signal, but it changes a public tool surface — a required field plus a 15-25 word instruction on all 17 toolsets — so this ships with context: false. Worth turning on as its own change once the basic numbers are landing.

The SDK is pre-1.0 (0.x), so event names and properties can change in minor releases — hence the exact-version pin.

Verification

  • bunx tsc --noEmit passes (what CI runs).
  • Smoke tested against a non-production PostHog project with an in-memory MCP client: $mcp_initialize, $mcp_tools_list, and both a successful and a failing $mcp_tool_call arrive with latency and error flag populated, and with context: false the tool schemas come back unmodified.
  • Redaction verified the same way with canary values planted in a created key, a credential values map, and a CDP URL: no $mcp_response on any event, no $mcp_parameters on the credential tools, parameters intact on manage_browsers, and no canary string anywhere in the captured events.
  • bun run build fails locally on this branch and on main alike (page data collection for /authorize without Clerk/Redis env), so that failure is unrelated.

@vercel

vercel Bot commented Jul 29, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
mcp Ready Ready Preview Jul 29, 2026 7:36pm
mcp (staging) Ready Ready Preview Jul 29, 2026 7:36pm

@socket-security

socket-security Bot commented Jul 29, 2026

Copy link
Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Addedposthog-node@​5.46.17210079100100
Added@​posthog/​mcp@​0.10.17910099100100

View full report

@masnwilliams
masnwilliams marked this pull request as ready for review July 29, 2026 19:11
Comment thread src/lib/mcp/analytics.ts
@masnwilliams
masnwilliams marked this pull request as ready for review July 29, 2026 19:27

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit b1a9ccf. Configure here.

Comment thread src/lib/mcp/analytics.ts
"manage_api_keys",
"manage_credentials",
"manage_proxies",
]);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Credential tool allowlist incomplete

High Severity

CREDENTIAL_TOOLS omits manage_auth_connections, whose fields map is the same arbitrary name-to-value pattern as manage_credentials values. Key-name redaction cannot cover those entries, so login/MFA submit parameters can still ship to PostHog in $mcp_parameters.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit b1a9ccf. Configure here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant