Istio sidecar startup races, and why your pods fail only sometimes

The symptom

A deployment succeeds most of the time. Occasionally a pod comes up and its first few outbound calls fail — connection refused, or a 503 with no corresponding entry in the destination service logs. The pod then settles and behaves normally. Restarting it usually makes the problem disappear, which is exactly what makes it hard to chase.

The failure rate tends to correlate with how quickly the application makes its first outbound request. Services that connect to a database, fetch configuration, or register with a discovery system during startup are affected far more often than services that wait for their first inbound request.

The mechanism

In sidecar mode, Istio adds two things to the pod. An init container (`istio-init`) installs iptables rules that redirect the pod network namespace traffic through the proxy, and a sidecar container (`istio-proxy`) runs Envoy itself.

Init containers run to completion before application containers start. That ordering is the problem: by the time your application container begins executing, the iptables rules are already in place and every outbound connection is being redirected to a port that Envoy may not yet be listening on. Kubernetes starts the application container and the proxy container in parallel, and nothing in the default configuration makes the application wait.

The window is short — usually a second or two while Envoy starts and receives its initial configuration from istiod — which is precisely why the failure is intermittent rather than consistent.

How to confirm it

The distinguishing signal is the timing relationship between the application failure and proxy readiness, not the error text itself.

• Compare container start timestamps in the pod status against the timestamp of the first application error. A failure occurring before the proxy reports ready is the signature.
• Query the proxy readiness endpoint on port 15021 (`/healthz/ready`) and note how long it takes to return success after container start.
• Check whether the failing calls appear in the sidecar access log at all. Requests that fail before Envoy is listening never reach it, so their absence from the proxy log is informative.
• Correlate with application startup logic: services making outbound calls in an init hook, constructor, or readiness-gated bootstrap are the usual candidates.

What to change

There are three broad options, and which one is right depends on the Kubernetes and Istio versions you are running.

• Native sidecar containers. Kubernetes added sidecar support via init containers with `restartPolicy: Always`, which makes the kubelet start the proxy before application containers and keep it running for the pod lifetime. Istio can use this when the feature is enabled and the cluster version supports it. This addresses the ordering problem structurally rather than by waiting, and is the preferred path where available.
• `holdApplicationUntilProxyStarts`. Istio can inject the proxy as the first container with a postStart hook that blocks until Envoy is ready, delaying application container start. It is settable mesh-wide in the proxy configuration or per-workload via the `proxy.istio.io/config` annotation. It costs a small amount of startup latency on every pod.
• Make the application tolerant. Retrying the first outbound connection with backoff is worth doing regardless, because it also covers transient failures that have nothing to do with the mesh. This is the only option that survives a misconfiguration of the other two.

The mirror image at shutdown

The same ordering problem occurs in reverse during termination, and it is easier to miss because the symptom lands on the client rather than the terminating pod. If Envoy stops before the application has finished draining in-flight requests, those requests fail. Istio exposes `terminationDrainDuration` to hold the proxy open, and an option to exit once active connections reach zero.

Kubernetes Jobs deserve a specific mention. A Job pod is considered complete when its containers exit, but a sidecar that runs indefinitely never exits, so the Job hangs. Historically this was handled by signalling the proxy to quit through the `pilot-agent` administrative endpoint once the workload finished. Native sidecar containers resolve this case as well, since the kubelet terminates them when the main containers complete.