When running Kubernetes on bare metal clusters, MetalLB provides load balancing functionality that cloud providers offer automatically. However, administrators frequently encounter issues where MetalLB fails to assign external IPs or the assigned IPs remain unreachable. This guide from PerLod Hosting explains why MetalLB external IP is not working and provides the best troubleshooting steps to resolve these issues.
For the complete setup guide, you can check the MetalLB Bare Metal Load Balancing tutorial, which covers installation and basic configuration.
How MetalLB Assigns External IPs?
MetalLB assigns external IP addresses to LoadBalancer services through a two-component system. The controller component allocates IP addresses from configured pools, while speaker components announce these IPs on the network using Layer 2 (ARP or NDP) or BGP protocols.
When you create a LoadBalancer service, the MetalLB controller watches for the new service and selects an available IP from an IPAddressPool resource.
The controller assigns this IP to the service's status field. After the assignment, speaker pods announce the IP on the network so external clients can reach it.
The process requires two custom resources:
1. IPAddressPool: Defines available IP ranges:
1apiVersion: metallb.io/v1beta12kind: IPAddressPool3metadata:4 name: production-pool5 namespace: metallb-system6spec:7 addresses:8 - 192.168.1.240-192.168.1.250
2. L2Advertisement: Tells MetalLB to announce IPs via ARP:
1apiVersion: metallb.io/v1beta12kind: L2Advertisement3metadata:4 name: l2-advertisement5 namespace: metallb-system6spec:7 ipAddressPools:8 - production-pool
Both resources are required. Creating only the IPAddressPool without L2Advertisement is the most common configuration mistake that causes unreachable IPs.
In Layer 2 mode, MetalLB speakers answer ARP requests for service IPs. When a device asks Who has 192.168.1.240?, MetalLB replies with the node's MAC address, so traffic routes to that node, and kube-proxy sends it to the correct pods.
Discover MetalLB External IP Issues
MetalLB failures usually happen in three ways: services stay pending, the IP address exists but cannot be reached, or a core component crashes.
1. External IP Stuck in Pending State: When services show pending in the EXTERNAL-IP column, MetalLB has not assigned an IP address.
Check the service status with the command below:
kubectl get svc -A | grep LoadBalancer
In the output, you will see:
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)default nginx-service LoadBalancer 10.96.123.45 <pending> 80:30123/TCP
This happens because:
- No Pool Created: You have not defined an IPAddressPool yet.
- Out of IPs: Every IP in your pool is already in use.
- Invalid IP Format: The IP range is written incorrectly.
- Controller Down: The MetalLB controller is not running.
- Wrong Namespace: The IP pool is restricted to a specific namespace that does not match your service.
You can check controller logs to identify the specific cause:
kubectl logs -n metallb-system -l component=controller --tail=50
Common error messages include:
- No available IPs: Pool is exhausted.
- No matching pool: No pool matches service requirements.
- Invalid CIDR notation: IP format errors.
To verify IPAddressPool exists, you can run:
kubectl get ipaddresspool -n metallb-systemkubectl describe ipaddresspool -n metallb-system
2. External IP Assigned But Not Reachable: This occurs when the service shows an external IP, but you cannot connect to it from the network.
Check if IP is assigned with the command below:
kubectl get svc nginx-service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)nginx-service LoadBalancer 10.96.123.45 192.168.1.240 80:30123/TCP
Testing from a client fails:
curl http://192.168.1.240
This happens because:
- Missing Advertisement: You created an IP pool but forgot the L2Advertisement resource to actually announce it.
- Subnet Mismatch: The load balancer IPs are not in the same network subnet as your cluster nodes.
- Strict ARP Disabled: You are using kube-proxy in IPVS mode but have not enabled strictARP.
- Nodes Excluded: Your nodes are labeled to explicitly ignore external load balancers.
- Permission Errors: The MetalLB speaker pods lack the permissions needed to reply to ARP requests.
Verify L2Advertisement exists with the following command:
kubectl get l2advertisement -n metallb-system
If missing, this is the problem.
Check speaker logs with the command below:
kubectl logs -n metallb-system -l component=speaker --tail=100
Look for announcement messages or errors like operation not permitted or failed to create ARP responder.
3. MetalLB Components Not Running: If MetalLB pods are not running, no IP assignments or announcements occur.
Check pod status with the command below:
kubectl get pods -n metallb-system -o wide
A healthy output looks like this:
NAME READY STATUS RESTARTS AGE NODEcontroller-7d4c8764f4-x9k8p 1/1 Running 0 2d worker-1speaker-2m9pm 1/1 Running 0 2d worker-1speaker-7m4qw 1/1 Running 0 2d worker-2
If pods show CrashLoopBackOff or ImagePullBackOff, check pod details with the commands below:
kubectl describe pod -n metallb-system <pod-name>kubectl logs -n metallb-system <pod-name>
This happens because:
- Image pull failures due to network issues.
- Insufficient resources on nodes.
- RBAC permission issues.
- CNI plugin conflicts.
How To Fix MetalLB External IP Failures?
Now that you have learned to identify MetalLB external IP failures, you can proceed to the following steps to fix the issues.
Fix Missing MetalLB L2Advertisement Configuration
In this problem, the external IP is assigned but unreachable, which means L2Advertisement is missing.
Check if the L2Advertisement exists with the command below:
kubectl get l2advertisement -n metallb-system
If it returns No resources found, you must create the L2Advertisement:
1apiVersion: metallb.io/v1beta12kind: L2Advertisement3metadata:4 name: default-l2-advertisement5 namespace: metallb-system6spec:7 ipAddressPools:8 - production-pool
Apply it with the command below:
kubectl apply -f l2advertisement.yaml
Verify it by using the commands below:
kubectl get l2advertisement -n metallb-systemkubectl describe l2advertisement -n metallb-system
Also check speaker logs for announcements:
kubectl logs -n metallb-system -l component=speaker | grep -i "announced"
Finally, test connectivity with the following commands:
ping 192.168.1.240curl http://192.168.1.240
Fix MetalLB IP Pool Configuration Errors
In this issue, services are stuck in a pending state due to pool configuration issues.
If it happens because of invalid IP format, you must use the correct format:
1addresses:2- 192.168.1.240-192.168.1.250 3- 192.168.1.0/24
If it happens because of the wrong subnet, check the node IPs:
kubectl get nodes -o wide
If nodes are on 192.168.4.0/24, MetalLB IPs must be in the same subnet:
1apiVersion: metallb.io/v1beta12kind: IPAddressPool3metadata:4 name: production-pool5 namespace: metallb-system6spec:7 addresses:8 - 192.168.4.240-192.168.4.250
If the problem happens because the IP pool is exhausted, count LoadBalancer services:
kubectl get svc -A | grep LoadBalancer | wc -l
If count equals your pool size, you must expand the range:
1apiVersion: metallb.io/v1beta12kind: IPAddressPool3metadata:4 name: production-pool5 namespace: metallb-system6spec:7 addresses:8 - 192.168.1.240-192.168.1.2509 - 192.168.1.160-192.168.1.170
Apply and verify it with the commands below:
kubectl apply -f ipaddresspool.yamlkubectl get ipaddresspool -n metallb-systemkubectl describe ipaddresspool -n metallb-system
Fix StrictARP Configuration for IPVS Mode
In this state, external IPs are unreachable when using kube-proxy in IPVS mode.
You can check the kube-proxy mode with the command below:
kubectl get configmap kube-proxy -n kube-system -o yaml | grep mode
If output shows ipvs mode, check strictARP:
kubectl get configmap kube-proxy -n kube-system -o yaml | grep strictARP
If strictARP is false or missing, enable it with:
kubectl edit configmap kube-proxy -n kube-system
1apiVersion: kubeproxy.config.k8s.io/v1alpha12kind: KubeProxyConfiguration3mode: "ipvs"4ipvs:5 strictARP: true
Restart kube-proxy pods and verify it with the commands below:
kubectl delete pods -n kube-system -l k8s-app=kube-proxykubectl get configmap kube-proxy -n kube-system -o yaml | grep strictARP
Fix MetalLB Node Label Exclusions
In this problem, nodes with an exclusion label prevent MetalLB from announcing IPs.
Check for the exclusion label with the commands below:
kubectl get nodes --show-labels | grep exclude-from-external-load-balancerskubectl describe node <node-name> | grep exclude-from-external-load-balancers
You can remove the exclusion label with the following command:
kubectl label node <node-name> node.kubernetes.io/exclude-from-external-load-balancers-
For all nodes, you can run:
kubectl label nodes --all node.kubernetes.io/exclude-from-external-load-balancers-
Verify the removal process is completed with the command below:
kubectl get nodes --show-labels | grep exclude-from-external-load-balancers
Fix MetalLB Network Routing and Subnet Issues
In this step, the problem is different subnets between nodes and MetalLB IPs.
You can check node IPs with the command below:
kubectl get nodes -o wide
Check the MetalLB pool with:
kubectl get ipaddresspool -n metallb-system -o yaml
If subnets do not match, you must update IPAddressPool to match the node subnet:
1apiVersion: metallb.io/v1beta12kind: IPAddressPool3metadata:4 name: production-pool5 namespace: metallb-system6spec:7 addresses:8 - 192.168.4.240-192.168.4.250
Ensure these IPs:
- Are not assigned to other devices.
- Are excluded from DHCP ranges.
- Are not gateway addresses.
Apply and test with the following commands:
kubectl apply -f ipaddresspool.yamlping 192.168.1.240curl http://192.168.1.240
Debug Issues with MetalLB Logs and Events
If your load balancer is not working, the logs will tell you why. You can use these commands to detect internal errors, verify ARP announcements, and track service events to identify exactly where the connection is failing.
Check controller logs with the command below:
kubectl logs -n metallb-system -l component=controller --tail=100
Look for:
- Assigned IP: Success
- No available IPs: Pool exhausted
- No matching pool: Configuration issue
Check speaker logs with the following command:
kubectl logs -n metallb-system -l component=speaker --tail=100
You must look for:
- serviceAnnounced: Success
- Failed to create ARP responder: Permission error
- Operation not permitted: Capability issue
You can check service events with the command below:
kubectl describe svc nginx-service
In the output, look for:
- IPAllocated: IP successfully assigned
- AllocationFailed: Error message
Use the commands below to check component status:
kubectl get pods -n metallb-systemkubectl get ipaddresspool -n metallb-systemkubectl get l2advertisement -n metallb-system
For network verification, you can use:
arp -n | grep 192.168.1.240ping 192.168.1.240 sudo tcpdump -i eth0 arp host 192.168.1.240 -n
Conclusion
MetalLB external IP failures on bare metal occur because of configuration errors, missing resources, or network incompatibilities. If your service stays pending, MetalLB cannot allocate an IP. Check that your IPAddressPool exists and uses valid ranges. If the IP is assigned but unreachable, MetalLB cannot announce it. Verify that the L2Advertisement resource exists, the IP matches your node subnet, and strictARP is enabled for IPVS mode.
For dedicated server environments running Kubernetes, MetalLB provides essential load-balancing capabilities that would otherwise require external hardware load balancers.
Always resolve issues by checking the controller and speaker logs and testing connectivity after every fix.
We hope you enjoy this MetalLB external IP error fixes guide. Subscribe to our X and Facebook channels to get the latest updates and articles.
