← Guides

Debugging an EKS LoadBalancer That Won't Go Healthy

KubernetesAWSEKSTerraformDevOps August 2026

The symptom

A LoadBalancer-type Service on EKS (a classic ELB, provisioned by the in-tree AWS cloud provider) was returning curl: (52) Empty reply from server, and aws elb describe-instance-health showed every registered instance as OutOfService.

Three unrelated problems turned out to be layered on top of each other. None of them alone would have been obvious from the symptom — the fix only became clear by checking each layer of the traffic path independently, from the Kubernetes Service outward to AWS networking.

The traffic path

Internet --> ELB (port 80) --> NodePort 31148 on a worker node --> pod (port 8080)

Every layer in that chain has its own way of silently dropping traffic. The debugging approach that worked: verify each hop independently, starting from the Kubernetes side (cheapest to check) and working outward to AWS.

Root cause 1 — Service selector didn’t match the pod’s labels

kubectl get endpoints hk-deployment
# NAME            ENDPOINTS   AGE
# hk-deployment   <none>      17h

An empty ENDPOINTS list means the Service’s selector matched zero pods. Kubernetes does exact key+value matching on labels — there’s no fuzzy or partial match:

# deployment.yml — pod template label
labels:
  app: hk

# service-lb.yml — selector (BEFORE)
spec:
  selector:
    app.kubernetes.io/name: hk   # different KEY, same value "hk"

app and app.kubernetes.io/name are two different keys. Even though both happened to equal "hk", the Service considered this zero matching pods — same as if the value were also different. Fix: make the selector key match the pod label key exactly (app: hk).

Gotcha: this is invisible unless you specifically check kubectl get endpoints <service>. kubectl get svc alone looks completely normal — Service, ClusterIP, and NodePort all show up fine even with zero working endpoints.

Root cause 2 — the ELB was only enabled in one Availability Zone

Even after fixing the selector, most targets stayed OutOfService. The actual reason code from AWS was specific and useful:

Instance is in the EC2 Availability Zone for which LoadBalancer
is not configured to route traffic to.

The Service’s subnet annotation only listed one subnet:

annotations:
  service.beta.kubernetes.io/aws-load-balancer-subnets: subnet-06675bda1a1539f1f

A subnet belongs to exactly one Availability Zone. A classic ELB only creates a load-balancer node in the AZ(s) of the subnets it’s given — registering an instance from a different AZ does not make the ELB route to it, even with cross-zone load balancing on. Cross-zone balancing spreads load across already-enabled AZs; it doesn’t add new ones.

Fix: list one subnet per AZ, comma-separated, in the same annotation:

service.beta.kubernetes.io/aws-load-balancer-subnets: subnet-06675bda1a1539f1f,subnet-09ffb20c4da788637,subnet-0b4a2635964d4983e

Gotcha: this annotation’s value is a plain string, not a YAML list — no square brackets. [subnet-a, subnet-b] would parse as an actual YAML array and break the annotation’s expected string type. Comma-separated, no brackets; spaces after the commas don’t seem to matter but there’s no reason to risk it.

Root cause 3 (a red herring, sort of) — the security group work wasn’t the fix

A lot of time went into adding a security group rule allowing port 31148 (the NodePort) from the ELB’s security group, on each worker group individually. It turned out to be unnecessary: the shared EKS cluster security group (attached to every node automatically by the Terraform EKS module) already had a broad rule:

Type: All traffic, All ports
Source: <the ELB's own security group>

Security group rules are additive across every SG attached to an interface — if any attached security group allows the traffic, it’s allowed, regardless of what the others say. That one existing rule already covered the specific NodePort rule the team spent time adding elsewhere.

Lesson, not really a gotcha: when a symptom looks like a networking problem, it’s tempting to fix the security groups first because they’re the most familiar knob to turn. Worth checking all security groups attached to the actual network interface (not just the obviously-named ones) before concluding a specific rule is missing — a shared/default rule elsewhere in the stack might already cover it.

Other gotchas that came up along the way

NodePort values are randomly assigned, not fixed. A LoadBalancer or NodePort Service without an explicit nodePort: field gets a random port from the cluster’s NodePort range (typically 30000–32767) the first time it’s created. If the Service is ever deleted and recreated (not just applyd again), it can get a different port — silently breaking any security group rule that references the old port number by value. Pin it explicitly if you want it stable:

ports:
  - port: 80
    targetPort: 8080
    nodePort: 31148   # otherwise this is random

git and infrastructure state are two separate systems. terraform apply and kubectl apply both act on whatever is in your local files right now — committed or not, pushed or not. It’s entirely possible to change real AWS/cluster state locally without a single commit existing anywhere, which is exactly how a teammate’s local terraform apply caused confusion earlier in this project: the change was real, but invisible to anyone looking at GitHub.

kubectl always reads live cluster state, not local files. kubectl get pod --show-labels and kubectl get svc -o jsonpath=... ask the API server directly — they reflect whatever was last applyd to the cluster, which can differ from what’s currently sitting in your working directory if you’ve edited a file since the last apply (in either direction).

A smart cd (zoxide, autojump, etc.) can silently jump you to the wrong directory. If your shell aliases cd to a fuzzy-jump tool (zoxide init --cmd cd), running cd /exact/path/that/no/longer/exists doesn’t error — it silently fuzzy-matches to the closest known directory instead, which might be a different clone of the same repo. Confirm you’re where you think you are with pwd and git rev-parse --show-toplevel before trusting output, especially after multiple clones of a repo have accumulated on disk.

ELB reason codes are worth reading literally. aws elb describe-instance-health doesn’t just say “unhealthy” — the description field gives the actual cause (Instance registration is still in progress, Instance is in the EC2 Availability Zone for which LoadBalancer is not configured to route traffic to, Instance has failed at least the UnhealthyThreshold number of health checks). Each of these points to a completely different fix; don’t treat “OutOfService” as one undifferentiated problem.

Debugging order that worked

  1. kubectl get endpoints <service> — confirms the Service has something to route to at all, before touching anything network-related.
  2. kubectl get svc <service> -o jsonpath='{.spec.selector}' vs kubectl get pod <pod> --show-labels — catches selector/label mismatches directly.
  3. aws elb describe-instance-health --load-balancer-name <name> — read the Description field, not just the State.
  4. aws elb describe-load-balancers — check AvailabilityZones/Subnets against the AZs your actual instances are running in.
  5. Only then, security groups — check every SG attached to the relevant network interface, not just the one with the obvious name.