Reading Kubernetes API throttling before it becomes an outage

The symptom

Commands that used to return instantly take seconds. Controllers and operators fall behind, reconciling minutes after the change that triggered them. Autoscaling reacts late. Node registration during a scale-up drags. Nothing is down, and dashboards showing node and pod health look normal.

On EKS the control plane is managed and scales on your behalf, which leads people to assume API capacity is not something they need to think about. Managed scaling does not exempt a cluster from the fairness and concurrency controls that protect the API server.

Two different throttles

The critical distinction is where the request is being delayed, because the two causes call for opposite responses.

Client-side throttling happens inside the calling process. Kubernetes clients carry a rate limiter with a queries-per-second and burst setting, and when a controller exceeds its own limit the client delays the request before it ever reaches the network. The API server never sees it and cannot report it. Historically these limiter defaults were low enough that a busy controller could throttle itself while the API server sat comfortably idle.

Server-side throttling is the API server protecting itself. API Priority and Fairness classifies incoming requests into flows, queues them by priority level, and rejects with HTTP 429 when a level is saturated. This is visible on the server and is a signal about total cluster load rather than about one client.

Telling them apart

Do not guess from latency alone. Each mechanism announces itself differently.

• Client-side throttling is logged by the client itself, in messages noting a wait due to client-side throttling and explicitly distinguishing it from priority and fairness. Raise client log verbosity if you do not see them.
• Server-side rejection appears as HTTP 429 responses in API server request metrics, and in the flow control metrics that count rejected requests and in-queue requests per priority level.
• On EKS, enable control plane logging to CloudWatch so the API server and audit logs are available. Audit logs are the fastest way to find which user agent is generating the request volume.
• If kubectl is slow but the flow control metrics show no rejections and no queueing, the constraint is on the client side.

What generates the load

Most severe cases trace back to a small number of patterns, and the offender is frequently a controller nobody was thinking about.

• Repeated full list calls on large collections instead of a watch-backed informer cache. This is the single most common cause and scales with cluster size.
• Listing without a field or label selector, then filtering client-side, which forces the API server to serialize objects the caller immediately discards.
• Reconcile loops that requeue aggressively, so error conditions produce a tight retry cycle against the API.
• High object counts, particularly custom resources with large specs or frequent status writes, since each status update is a write against the API server.
• Many clients waking on the same interval, producing synchronized bursts rather than an even request rate.

What to change

Reduce demand before raising limits. Raising a client rate limit on a controller that is listing all pods every few seconds converts a client-side delay into server-side pressure, which is a worse failure because it now affects every other client on the cluster.

Move list-heavy code onto shared informers so it reads from a local cache maintained by a single watch. Add selectors so the server filters instead of the client. Back off on error rather than requeueing immediately.

Once demand is genuinely reduced and a specific workload still needs more throughput, adjust its client rate limit deliberately, and consider a dedicated flow schema and priority level so a busy controller cannot starve interactive requests. Treat priority level configuration as capacity planning, not as a way to silence 429s.