agentgateway Troubleshooting: MCP Routing, TLS, Auth & OpenTelemetry

Diagnosing agentgateway failures gets fast and reliable once you split every symptom into transport, identity, routing, or telemetry buckets, then run config checks on each layer. This agentgateway troubleshooting guide focuses on failed MCP routes, upstream TLS resets, bad tokens, protocol translation errors, missing OTel spans, and latency spikes.

agentgateway sits between your clients and your backends, including LLM providers, MCP servers, or regular HTTP services, and applies routing, TLS, authentication, and telemetry policies on every request. Because it touches every layer of the request path, most agentgateway troubleshooting work is about isolating which layer broke:

• Network connection.
• Certificate.
• Token.
• Protocol translation between the client and the backend.

Every agentgateway pod runs an admin server on port 15000 with endpoints such as /config_dump, /logging, /debug/trace, and /debug/pprof, which are the backbone of any agentgateway troubleshooting session.

Check this first, before logs or curl. It shows what the proxy actually loaded, not what you wrote in the config file:

When an MCP client gets connection refused, 404, or "no matching route" errors, the problem is a mismatch between the Gateway, the HTTPRoute, and the backend selector, not the MCP server itself.

Start every route-related agentgateway troubleshooting task by checking resource status before diving into packet-level debugging.

1. Confirm the control plane and proxy pods are actually running:

2. Verify the Gateway is Accepted and Programmed:

3. Check the HTTPRoute for Accepted and ResolvedRefs conditions:

4. Dump the actual loaded config to see what the proxy really matched:

5. Inspect per-backend endpoint health and request counts:

6. Trace a single request end to end to see which route and backend were selected:

These checks usually reveal one of three problems:

• Two HTTPRoutes with the same or no path matcher, where the empty one defaults to "/" and blocks the other.
• A route pointing at the wrong Gateway.
• A typo in the backend name.

This one step fixes most agentgateway troubleshooting tickets, because the route usually isn't broken. It's just hidden behind another rule.

Connection resets and "certificate verify failed" errors on the backend side come from the backendTLS block in your route policy. This is a common agentgateway troubleshooting case because the client-facing side can look perfectly healthy while the backend connection still fails.

By default, agentgateway sends plain HTTP to backends. To use TLS, you have to turn on backendTLS. Once enabled, it trusts the system's CA certificates and automatically picks the SNI from the backend's hostname.

If you're using a private or internal CA, point to it directly instead of relying on the system's default trust store:

First, test the upstream on its own with openssl, outside of agentgateway, to see if the problem is really there:

Read the output for expiry dates, the certificate chain, and the subject name. A mismatch between the SNI agentgateway sends and the certificate's subjectAltNames is the most frequent cause flagged during agentgateway troubleshooting of upstream TLS resets.

Also, confirm with curl that the TLS handshake succeeds before blaming agentgateway:

If curl works but agentgateway still resets the connection, try setting the SNI manually. This rules out a hostname mismatch:

Only while testing, turn off certificate verification to check if TLS trust is the real blocker. Never leave this on in production:

Authentication failures show up as 401 or 403 responses, and agentgateway logs the exact reason in plain text, which makes this stage of agentgateway troubleshooting faster than most people expect.

JWT authentication in agentgateway needs an issuer, an audience list, and a jwks source, and it runs in one of three modes:

• Strict
• Optional
• Permissive

Optional mode still checks a token if you send one, but it lets requests through with no token at all. If you want every request without a token to be rejected, make sure the mode is set to strict.

Run these checks in order during agentgateway troubleshooting of auth issues:

1. Watch the proxy logs and compare the error message with and without a token. This tells you if the gateway or the backend is the one rejecting the call:

2. Check that the JWT header has kid and exp values. agentgateway needs both to validate the token against a JWKS.

3. Curl the JWKS endpoint directly. This checks if the key set is unreachable or was recently rotated:

4. If you use backendAuth to attach an outbound token to the backend request, confirm the key and location match what the backend expects:

By default, agentgateway removes the client's original token before sending the request to the backend. If the backend needs that original token instead of a new one, use passthrough instead of key.

A common mistake in agentgateway troubleshooting is confusing jwtAuth, which checks the client's token, with backendAuth, which attaches a token agentgateway sends onward.

agentgateway frequently translates between transports, such as HTTP clients calling stdio-based MCP servers, or REST calls being converted into provider-specific LLM API formats.

When responses come back malformed or in the wrong shape, protocol translation is the usual suspect in this part of agentgateway troubleshooting.

Confirm the exact transport expected by the client with the MCP inspector CLI:

If this returns clean JSON but your real client still fails, check the mcp-protocol-version header and the transport type your client uses, stdio, streamable HTTP, or SSE. Many MCP clients silently ignore a mismatch here instead of showing an error.

Before blaming the protocol logic, test a plain TLS handshake on the HTTPS listener to rule out the transport layer:

If you see "Not Acceptable: Client must accept text/event-stream," that's actually good news. It means TLS and routing are working fine, and the only remaining issue is how the client negotiates the MCP protocol.

Missing spans usually mean traces never left agentgateway, not that your collector is broken. So the first thing to check in this stage of agentgateway troubleshooting is the OTLP export config.

agentgateway needs frontendPolicies.tracing configured with a reachable OTLP gRPC host:

Only use randomSampling: true in development, so every request gets traced. In production, use a decimal like "0.1" instead, which samples 10 percent of traffic and keeps overhead low.

Before blaming agentgateway, check that the OTLP endpoint is actually reachable:

Run a quick local Jaeger instance to check if the full export actually works:

Then check http://localhost:16686 for incoming traces. For production, route traces through an OpenTelemetry Collector and add a debug exporter with verbosity: detailed to its pipeline so you can see exactly what the collector receives:

If spans still don't show up, check that the collector's receiver is bound to 0.0.0.0:4317, not just localhost. Container networking often blocks loopback-only bindings. For more on collector-side export issues once traces leave agentgateway, see this guide on deploying and troubleshooting SigNoz as your tracing backend.

High latency in agentgateway is usually one of a slow upstream, TLS handshake overhead on every request, or a policy such as JWT validation adding processing time.

Capture a live per-request trace to see exactly where time is spent:

Also, inspect backend endpoint health and per-backend latency directly:

This output has a LATENCY column for each backend, so you can quickly spot if one upstream is slowing everything down. This saves a lot of guesswork in agentgateway troubleshooting. If all backends are slow, start by raising the log level to check if a specific policy is causing the delay:

Then capture a CPU profile to check whether the proxy itself is the bottleneck:

Also capture a heap profile if memory pressure, not CPU, seems to be the cause of slow responses:

Keep these commands on hand for agentgateway troubleshooting. Together, they cover most of what's described above.

Confirm the control plane and proxy are running:

Confirm routing objects are accepted:

See the real loaded config:

Check backend health and latency:

Trace one request end to end:

Verify upstream certificates:

Test connectivity and TLS outside agentgateway:

Read plain-text auth and routing errors:

Raise proxy log verbosity live:

Capture a CPU profile:

Shared hosting and local NAT setups often hide the real cause of connection resets and TLS failures behind double NAT, port conflicts, or noisy-neighbor traffic, which makes any agentgateway troubleshooting session unreliable no matter how many commands you run.

If it keeps working one moment and failing the next, with no clear pattern, the problem is likely your network, not your config.

Moving your gateway to a clean Linux VPS gives you your own public IP, predictable routing, and full control over firewall rules. This clears out a whole layer of noise before you even open a debug endpoint.

Most agentgateway troubleshooting depends on answering this question: is this a transport, identity, routing, or telemetry problem? Once you isolate the layer with the checklist above, the fix is usually a one-line config change, a certificate refresh, or a corrected route matcher.

We hope you enjoy this guide.

For deeper background on OpenTelemetry Collector deployment patterns, you can check the OpenTelemetry documentation on agent-to-gateway deployment.

To install the agentgateway single-binary package, you can check this guide on agentgateway setup on Ubuntu.