Diagnosing agentgateway failures gets fast and reliable once you split every symptom into transport, identity, routing, or telemetry buckets, then run config checks on each layer. This agentgateway troubleshooting guide focuses on failed MCP routes, upstream TLS resets, bad tokens, protocol translation errors, missing OTel spans, and latency spikes.

How agentgateway Handles Requests

agentgateway sits between your clients and your backends, including LLM providers, MCP servers, or regular HTTP services, and applies routing, TLS, authentication, and telemetry policies on every request. Because it touches every layer of the request path, most agentgateway troubleshooting work is about isolating which layer broke:

• Network connection.
• Certificate.
• Token.
• Protocol translation between the client and the backend.

Every agentgateway pod runs an admin server on port 15000 with endpoints such as /config_dump, /logging, /debug/trace, and /debug/pprof, which are the backbone of any agentgateway troubleshooting session.

Check this first, before logs or curl. It shows what the proxy actually loaded, not what you wrote in the config file:

kubectl port-forward deploy/agentgateway-proxy -n agentgateway-system 15000 &
curl http://localhost:15000/config_dump

Fix Failed MCP Routes in agentgateway

When an MCP client gets connection refused, 404, or “no matching route” errors, the problem is a mismatch between the Gateway, the HTTPRoute, and the backend selector, not the MCP server itself.

Start every route-related agentgateway troubleshooting task by checking resource status before diving into packet-level debugging.

1. Confirm the control plane and proxy pods are actually running:

kubectl get pods -n agentgateway-system

2. Verify the Gateway is Accepted and Programmed:

kubectl get gateway agentgateway-proxy -n agentgateway-system -o yaml

3. Check the HTTPRoute for Accepted and ResolvedRefs conditions:

kubectl get httproute mcp-github -n agentgateway-system -o yaml

4. Dump the actual loaded config to see what the proxy really matched:

agctl proxy config all gateway/agentgateway-proxy -n agentgateway-system -o yaml

5. Inspect per-backend endpoint health and request counts:

agctl proxy config backends gateway/agentgateway-proxy -n agentgateway-system

6. Trace a single request end to end to see which route and backend were selected:

agctl proxy trace gateway/agentgateway-proxy -n agentgateway-system --port 8080 -- http://mcp-github.example.com/

These checks usually reveal one of three problems:

• Two HTTPRoutes with the same or no path matcher, where the empty one defaults to “/” and blocks the other.
• A route pointing at the wrong Gateway.
• A typo in the backend name.

This one step fixes most agentgateway troubleshooting tickets, because the route usually isn’t broken. It’s just hidden behind another rule.

agentgateway Troubleshooting for Upstream TLS Errors

Connection resets and “certificate verify failed” errors on the backend side come from the backendTLS block in your route policy. This is a common agentgateway troubleshooting case because the client-facing side can look perfectly healthy while the backend connection still fails.

By default, agentgateway sends plain HTTP to backends. To use TLS, you have to turn on backendTLS. Once enabled, it trusts the system’s CA certificates and automatically picks the SNI from the backend’s hostname.

binds:
- port: 3000
  listeners:
  - routes:
    - backends:
      - mcp:
          targets:
          - name: secure-upstream
            http:
              host: api.example.com
              port: 443
        policies:
          backendTLS:
            tls: {}

If you’re using a private or internal CA, point to it directly instead of relying on the system’s default trust store:

policies:
  backendTLS:
    tls:
      caCertificateRefs:
      - name: my-ca-configmap

First, test the upstream on its own with openssl, outside of agentgateway, to see if the problem is really there:

openssl s_client -connect backend.example.com:443 -servername backend.example.com -showcerts

Read the output for expiry dates, the certificate chain, and the subject name. A mismatch between the SNI agentgateway sends and the certificate’s subjectAltNames is the most frequent cause flagged during agentgateway troubleshooting of upstream TLS resets.

Also, confirm with curl that the TLS handshake succeeds before blaming agentgateway:

curl -v https://backend.example.com/health

If curl works but agentgateway still resets the connection, try setting the SNI manually. This rules out a hostname mismatch:

policies:
  backendTLS:
    tls:
      sni: api.internal.example.com

Only while testing, turn off certificate verification to check if TLS trust is the real blocker. Never leave this on in production:

policies:
  backendTLS:
    tls:
      insecureSkipVerify: All

Diagnose Authentication and Invalid Tokens in agentgateway

Authentication failures show up as 401 or 403 responses, and agentgateway logs the exact reason in plain text, which makes this stage of agentgateway troubleshooting faster than most people expect.

JWT authentication in agentgateway needs an issuer, an audience list, and a jwks source, and it runs in one of three modes:

• Strict
• Optional
• Permissive

llm:
  policies:
    jwtAuth:
      mode: strict
      issuer: agentgateway.dev
      audiences: [test.agentgateway.dev]
      jwks:
        file: ./manifests/jwt/pub-key
  models:
  - name: "*"
    provider: openAI
    params:
      apiKey: "$OPENAI_API_KEY"

Optional mode still checks a token if you send one, but it lets requests through with no token at all. If you want every request without a token to be rejected, make sure the mode is set to strict.

Run these checks in order during agentgateway troubleshooting of auth issues:

1. Watch the proxy logs and compare the error message with and without a token. This tells you if the gateway or the backend is the one rejecting the call:

kubectl logs -n agentgateway-system deploy/agentgateway-proxy -f

2. Check that the JWT header has kid and exp values. agentgateway needs both to validate the token against a JWKS.

3. Curl the JWKS endpoint directly. This checks if the key set is unreachable or was recently rotated:

curl -s https://your-idp.example.com/.well-known/jwks.json

4. If you use backendAuth to attach an outbound token to the backend request, confirm the key and location match what the backend expects:

backendAuth:
  key:
    value: $MY_API_KEY
    location:
      header:
        name: authorization
        prefix: "Bearer "

By default, agentgateway removes the client’s original token before sending the request to the backend. If the backend needs that original token instead of a new one, use passthrough instead of key.

A common mistake in agentgateway troubleshooting is confusing jwtAuth, which checks the client’s token, with backendAuth, which attaches a token agentgateway sends onward.

Solve agentgateway Protocol Translation Issues

agentgateway frequently translates between transports, such as HTTP clients calling stdio-based MCP servers, or REST calls being converted into provider-specific LLM API formats.

When responses come back malformed or in the wrong shape, protocol translation is the usual suspect in this part of agentgateway troubleshooting.

Confirm the exact transport expected by the client with the MCP inspector CLI:

npx @modelcontextprotocol/inspector-cli \
  --cli http://localhost:3000/mcp \
  --transport http \
  --header "mcp-protocol-version: 2024-11-05" \
  --method tools/call \
  --tool-name get_me

If this returns clean JSON but your real client still fails, check the mcp-protocol-version header and the transport type your client uses, stdio, streamable HTTP, or SSE. Many MCP clients silently ignore a mismatch here instead of showing an error.

Before blaming the protocol logic, test a plain TLS handshake on the HTTPS listener to rule out the transport layer:

curl https://localhost:3000 -k

If you see “Not Acceptable: Client must accept text/event-stream,” that’s actually good news. It means TLS and routing are working fine, and the only remaining issue is how the client negotiates the MCP protocol.

Recover Lost Traces in agentgateway

Missing spans usually mean traces never left agentgateway, not that your collector is broken. So the first thing to check in this stage of agentgateway troubleshooting is the OTLP export config.

agentgateway needs frontendPolicies.tracing configured with a reachable OTLP gRPC host:

frontendPolicies:
  tracing:
    host: localhost:4317
    randomSampling: true

Only use randomSampling: true in development, so every request gets traced. In production, use a decimal like “0.1” instead, which samples 10 percent of traffic and keeps overhead low.

Before blaming agentgateway, check that the OTLP endpoint is actually reachable:

curl -v telnet://localhost:4317

Run a quick local Jaeger instance to check if the full export actually works:

docker run -d --name jaeger \
  -p 16686:16686 \
  -p 4317:4317 \
  jaegertracing/all-in-one:latest

Then check http://localhost:16686 for incoming traces. For production, route traces through an OpenTelemetry Collector and add a debug exporter with verbosity: detailed to its pipeline so you can see exactly what the collector receives:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
exporters:
  debug:
    verbosity: detailed
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true
service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [debug, otlp/jaeger]

If spans still don’t show up, check that the collector’s receiver is bound to 0.0.0.0:4317, not just localhost. Container networking often blocks loopback-only bindings. For more on collector-side export issues once traces leave agentgateway, see this guide on deploying and troubleshooting SigNoz as your tracing backend.

Reduce High Latency in agentgateway

High latency in agentgateway is usually one of a slow upstream, TLS handshake overhead on every request, or a policy such as JWT validation adding processing time.

Capture a live per-request trace to see exactly where time is spent:

agctl proxy trace gateway/agentgateway-proxy -n agentgateway-system --port 8080 -- http://backend.example.com/

Also, inspect backend endpoint health and per-backend latency directly:

agctl proxy config backends gateway/agentgateway-proxy -n agentgateway-system

This output has a LATENCY column for each backend, so you can quickly spot if one upstream is slowing everything down. This saves a lot of guesswork in agentgateway troubleshooting. If all backends are slow, start by raising the log level to check if a specific policy is causing the delay:

agctl proxy log gateway/agentgateway-proxy -n agentgateway-system --level debug

Then capture a CPU profile to check whether the proxy itself is the bottleneck:

curl -o cpu.pprof "http://localhost:15000/debug/pprof/profile?seconds=30"
go tool pprof -http=: cpu.pprof

Also capture a heap profile if memory pressure, not CPU, seems to be the cause of slow responses:

curl -o heap.pprof http://localhost:15000/debug/pprof/heap
go tool pprof -http=: heap.pprof

Quick agentgateway Diagnostic Checklist

Keep these commands on hand for agentgateway troubleshooting. Together, they cover most of what’s described above.

Confirm the control plane and proxy are running:

kubectl get pods -n agentgateway-system

Confirm routing objects are accepted:

kubectl get gateway,httproute -A

See the real loaded config:

agctl proxy config all gateway/<name> -n <namespace> -o yaml

Check backend health and latency:

agctl proxy config backends gateway/<name> -n <namespace>

Trace one request end to end:

agctl proxy trace gateway/<name> -n <namespace> --port <port> -- <url>

Verify upstream certificates:

openssl s_client -connect host:443 -showcerts

Test connectivity and TLS outside agentgateway:

curl -v https://host/path

Read plain-text auth and routing errors:

kubectl logs -n agentgateway-system deploy/agentgateway-proxy -f

Raise proxy log verbosity live:

agctl proxy log gateway/<name> -n <namespace> --level debug

Capture a CPU profile:

curl -o cpu.pprof "http://localhost:15000/debug/pprof/profile?seconds=30"

Move to a Clean VPS When Networking Gets Unreliable

Shared hosting and local NAT setups often hide the real cause of connection resets and TLS failures behind double NAT, port conflicts, or noisy-neighbor traffic, which makes any agentgateway troubleshooting session unreliable no matter how many commands you run.

If it keeps working one moment and failing the next, with no clear pattern, the problem is likely your network, not your config.

Moving your gateway to a clean Linux VPS gives you your own public IP, predictable routing, and full control over firewall rules. This clears out a whole layer of noise before you even open a debug endpoint.

Conclusion

Most agentgateway troubleshooting depends on answering this question: is this a transport, identity, routing, or telemetry problem? Once you isolate the layer with the checklist above, the fix is usually a one-line config change, a certificate refresh, or a corrected route matcher.

We hope you enjoy this guide.

For deeper background on OpenTelemetry Collector deployment patterns, you can check the OpenTelemetry documentation on agent-to-gateway deployment.

To install the agentgateway single-binary package, you can check this guide on agentgateway setup on Ubuntu.

FAQs