Kubernetes Gateway API: Fix Routes, ReferenceGrant, and 404 Errors

Gateway API is the standard way to manage traffic in Kubernetes, but it has more moving parts than the old Ingress object. When a route stops working, you have to check the Gateway, the Route, the ReferenceGrant, and the backend Service. This guide gives you a simple runbook for Kubernetes Gateway API troubleshooting, so you always know which layer is broken before you touch a single manifest.

Before you start Kubernetes Gateway API troubleshooting, make sure that Gateway API CRDs are installed correctly. If you are new to Gateway API and migrating from Ingress, you can check this guide on Kubernetes Ingress to Gateway API.

Check that the CRDs are set up correctly with the command below:

Traffic passes through four layers, in this order:

• GatewayClass: Picks the controller, for example, nginx or cilium.
• Gateway: Opens a listener on a port and protocol.
• Route (HTTPRoute, TLSRoute, TCPRoute, UDPRoute, GRPCRoute): Matches traffic and sends it to a backend.
• ReferenceGrant: Needed only when a Route in one namespace points to a Gateway or Service in another namespace.

Each object shows its health in status.conditions. The objects tell you what's wrong. The three conditions you'll use most are Accepted, Programmed, and ResolvedRefs.

• Accepted = True means the object's configuration is valid and the controller has taken it.
• Programmed = True means the controller has pushed the configuration into the proxy.
• ResolvedRefs = True means every reference inside the object was found and allowed.

If any of these is False, the reason and message fields explain exactly why. This is the core idea behind good Kubernetes Gateway API troubleshooting. Always read the condition before you edit the YAML.

This is the one simple step that fixes most problems quickly. Every time, run this command first:

Look at the Status.Conditions block and the Status.Listeners block. Then, check the route:

Read Status.Parents, which lists a condition set for each Gateway the route is attached to. This tells you if the problem is at the Gateway level or the Route level.

This means the Gateway controller rejected the Route before even trying to program it. Common causes include:

• The parentRefs field points to a Gateway that does not exist, or is in a namespace the Route cannot see without a ReferenceGrant.
• The Route's hostname does not match any hostname allowed by the Gateway listener.
• The Gateway listener does not allow this Route kind, for example, an HTTPRoute pointed at a TCP listener.

Run the commands below to check:

To fix the issue, you must match the parentRefs.name, sectionName, and hostname exactly to what the Gateway listener allows. If the Route and Gateway are in different namespaces, add the correct kinds to the listener's allowedRoutes field.

This means the Route passed validation, but the actual proxy configuration was never applied. Traffic will not flow even though everything looks fine. Common causes include:

• The backend Service has zero ready endpoints, pods are crashing, or not labeled correctly.
• The controller pod itself is not running or is stuck resolving.
• A conflicting Route on the same listener has a higher-priority match.

To check for this issue, run the commands below:

If there are no endpoints, you must fix the pod or the Service selector first. If the controller pod is crashing, check its logs. This is usually a CRD version mismatch.

This means the Route is technically valid, but it is attached to the wrong Gateway or listener, so traffic goes to the wrong place or nowhere at all.

• Confirm parentRefs.name matches the Gateway name, and parentRefs.namespace is set if the Gateway lives elsewhere.
• If the Gateway has multiple listeners, always set sectionName to the listener name you want. Leaving it blank attaches to every compatible listener, which can cause unexpected matches.
• Check Status.Parents[].parentRef on the Route. This reflects exactly which Gateway and listener the controller thinks it is attached to.

Comparing the output with what you meant to write is the fastest way to catch a typo.

This means a Route in namespace A is trying to reach a Gateway, Service, or Secret in namespace B, but there is no permission slip for that connection.

Gateway API needs a ReferenceGrant every time a Route points to something in another namespace. The one exception is when a Route attaches to a Gateway in another namespace. That's allowed by the Gateway's own allowedRoutes field instead. If the ReferenceGrant is missing, the Route's ResolvedRefs condition turns False, with the reason RefNotPermitted.

Example fix: Allow HTTPRoutes in the team-a namespace to reference Services in team-b:

The ReferenceGrant must live in the same namespace as the target resource, not the namespace making the request. This is the most common mistake when troubleshooting cross-namespace Kubernetes Gateway API issues.

To verify it worked correctly, run the commands below:

This means HTTPS or TLSRoute traffic fails, drops, or serves the wrong certificate. Common causes include:

• The Secret holding the TLS certificate is in a different namespace than the Gateway, with no ReferenceGrant allowing the Gateway to read it.
• The Secret is not of type kubernetes.io/tls, or is missing tls.crt / tls.key.
• The listener's tls.mode is set to Passthrough but a Route is still trying to terminate TLS at the Gateway.

Run the above commands and look for the ResolvedRefs condition on the listener status.

If you see the reason InvalidCertificateRef, it means the Secret wasn't found, or the Gateway can't read it. Also note that TLSRoute became stable as v1 in Gateway API v1.5. If your controller still uses the old v1alpha2 version, update the controller first before you keep debugging.

This means the Route is accepted and programmed, but requests fail because the backend reference itself is wrong. Common causes include:

• The Service name or port in backendRefs does not exist.
• The Service exists but has no matching port name or number.
• The backend is in another namespace without a ReferenceGrant.

A ResolvedRefs = False with reason BackendNotFound or InvalidKind points directly at the backendRefs block. 

You must fix the name or port, or add the missing ReferenceGrant, then re-check the condition. It updates within a few seconds.

This means TCP or UDP traffic is not reaching the backend, even though HTTP routes on the same Gateway work fine.

You must check that both your CRDs and your controller support the new v1 version, not just the old one. As more teams add TCP and UDP routing, this version mismatch is becoming a common problem to check for. Common causes include:

• The Gateway listener protocol does not match. A TCPRoute cannot attach to a protocol: HTTP listener.
• sectionName is missing, and the route silently attaches to the wrong TCP listener when more than one exists.
• The controller does not yet support TCPRoute/UDPRoute at v1.

Here is an example listener and route:

Check status the same way as HTTPRoute:

Even when every Gateway API object shows healthy conditions, you can still get a 404 or 503 at the client. This is where Kubernetes Gateway API troubleshooting shifts from checking the API objects to checking the actual proxy behavior.

404 Not Found usually means:

• The hostname or path doesn't match any Route rule. Check for typos or a missing trailing slash in PathPrefix.
• The Route is Accepted but not Programmed yet. The proxy hasn't loaded the config. Wait a few seconds, then check the controller logs if it's still failing.
• With NGINX Gateway Fabric or Envoy-based controllers, a 404 means the request reached the proxy but matched no route. This is a proxy issue, not a Kubernetes issue.

503 Service Unavailable usually means:

• The backend Service has no ready pods. This happens when all pods fail their readiness probe.
• The controller's connection to the backend timed out. Check both the controller logs and the pod logs together.
• With Istio or Cilium mesh setups, a 503 can come from an mTLS policy blocking the connection, not from the Gateway API itself.

Commands to check across controllers:

Always check kubectl get events. Many controllers post an easy-to-read message the moment a Route or Gateway condition changes. This makes Kubernetes Gateway API troubleshooting much faster.

Here is a quick table you can check anytime. It shows the most common conditions, what a False status means, and where to look to fix it.

The best way to get good at Kubernetes Gateway API troubleshooting is practice. Break things on purpose in a safe test environment. Set up a new cluster on a stable dedicated server, install the Gateway API CRDs, and try each problem above. Soon, reading status conditions will feel easy and natural.

Most Gateway API problems come from a wrong parentRefs, a missing ReferenceGrant, a bad backend reference, or a controller that hasn't updated yet. Check the Accepted, Programmed, and ResolvedRefs conditions first, before you change anything. This helps you find the broken layer in minutes.

For more detailed information, check the Gateway API Official Guides.