Introduction to CrashLoopBackOff: The Most Common Kubernetes Error

CrashLoopBackOff is arguably the most frequently encountered and frustrating error state in Kubernetes, affecting both newcomers deploying their first Hello World applications and experienced SREs managing large-scale production clusters. When you see a pod stuck in CrashLoopBackOff status with steadily increasing restart counts, it means your container is caught in an infinite loop of failures: the container starts, immediately crashes or exits with an error, Kubernetes automatically restarts it following its self-healing principles, the container crashes again for the same reason, Kubernetes waits progressively longer before each restart attempt (exponential backoff: 10 seconds, 20 seconds, 40 seconds, 80 seconds, up to maximum 5 minutes), and this cycle continues indefinitely until you intervene and fix the underlying cause preventing the container from running successfully.

Understanding CrashLoopBackOff is critical because it's not actually a single specific error it's a symptom indicating that your container cannot run successfully. Engineers searching for what is CrashLoopBackOff often expect a single fix, but the actual root cause could be any of dozens of different problems ranging from simple configuration mistakes like missing environment variables to complex issues like application code bugs, database connection failures, insufficient memory allocation causing OOMKills, misconfigured health probes that incorrectly determine the application is unhealthy, missing dependencies like ConfigMaps or Secrets that the application requires to start, permission issues preventing the application from reading configuration files or writing logs, network problems preventing connections to required backend services, or resource constraints where the container needs more CPU or memory than allocated to function properly.

This comprehensive guide provides everything you need to understand, diagnose, and fix CrashLoopBackOff errors efficiently, covering: what CrashLoopBackOff actually means at a technical level and how Kubernetes' restart policy and exponential backoff work internally, the 15 most common root causes of CrashLoopBackOff with real-world examples and specific indicators for each, systematic debugging methodology using kubectl commands to identify root causes quickly, understanding container exit codes which provide crucial clues (exit code 0 = success, 1 = application error, 137 = OOMKilled, 143 = terminated by SIGTERM), analyzing pod events and container logs to find error messages explaining failures, fixing configuration issues like missing ConfigMaps or Secrets, resolving application startup failures and dependency problems, handling resource constraints and OOMKills, troubleshooting misconfigured liveness and readiness probes, and how an AI SRE platform such as Alert6 automates the investigation: correlating the related alerts into one incident, checking logs, events, telemetry and recent changes, identifying the probable root cause, and proposing a fix with commands and verification steps for your team to approve.

Get the root cause with the page, not an hour after it.

Alert6 correlates the alerts, investigates the evidence and proposes a fix your team approves. Read-only investigation by default.

Book a demo →

By mastering CrashLoopBackOff troubleshooting through this guide, you'll be able to diagnose and resolve pod crashes in minutes instead of hours, understand the systematic debugging process that works for any Kubernetes CrashLoopBackOff scenario, recognize common patterns and their solutions immediately, prevent CrashLoopBackOff through better configuration and testing practices, and use AI-powered tools to automate most of the investigation.

What is CrashLoopBackOff? Technical Explanation

The Crash-Restart Loop Explained

So, what is CrashLoopBackOff at the mechanical level? It is a pod status indicating Kubernetes is caught in a restart loop. Here's exactly what happens in the cycle:

  1. Container Starts: Kubernetes creates container from image, starts the main process
  2. Container Crashes: Process exits immediately (or shortly after starting) with non-zero exit code indicating failure
  3. Kubernetes Detects Exit: kubelet monitoring the container sees it terminated
  4. Restart Policy Triggers: restartPolicy: Always (default for Deployments) means Kubernetes will restart
  5. First Restart (10s delay): Kubernetes waits 10 seconds, then restarts container
  6. Second Crash: Container starts and crashes again (same problem, same failure)
  7. Second Restart (20s delay): Kubernetes waits 20 seconds (doubled), restarts again
  8. Third Crash and Restart (40s delay): Pattern continues with exponentially increasing delays
  9. BackOff Increases: Delays: 10s → 20s → 40s → 80s → 160s → capped at 300s (5 minutes)
  10. CrashLoopBackOff Status: After multiple failures, pod status shows CrashLoopBackOff

Why "BackOff"? Kubernetes implements exponential backoff to avoid overwhelming the system with rapid restart attempts. If a container crashes due to a problem that won't resolve automatically (like missing configuration), restarting it every second wastes resources and creates unnecessary API server load. The backoff gives time for manual intervention.

Checking CrashLoopBackOff Status

# List pods with status
kubectl get pods

# Output shows:
# NAME                    READY   STATUS             RESTARTS   AGE
# frontend-7d9f8b-xyz     0/1     CrashLoopBackOff   5          10m

# READY 0/1: 0 containers ready out of 1 total
# STATUS: CrashLoopBackOff
# RESTARTS 5: Container has crashed and restarted 5 times
# AGE 10m: Pod created 10 minutes ago

Understanding Restart Policy

CrashLoopBackOff behavior depends on pod's restartPolicy:

  • Always (default for Deployments, StatefulSets, DaemonSets): Restart on any exit, even if exit code 0
  • OnFailure (common for Jobs): Restart only if exit code non-zero (failure)
  • Never (rare): Don't restart regardless of exit code

Deployments use Always, so containers that crash (exit code != 0) are restarted indefinitely until they succeed or you delete the pod.

The 15 Most Common Causes of CrashLoopBackOff

Cause 1: Application Code Errors and Exceptions

Symptoms: Container starts but crashes immediately with exit code 1

Common scenarios:

  • Unhandled exceptions in application startup code (Python ImportError, Node.js module not found, Java ClassNotFoundException)
  • Application panics or crashes during initialization (Go panic, Rust panic)
  • Syntax errors in configuration files that application parses on startup
  • Missing required command-line arguments or flags

How to diagnose:

# Get logs from crashed container (CRITICAL: use --previous)
kubectl logs payment-service-7d9f8b-xyz --previous

# Common error patterns to look for:
# Python: "ModuleNotFoundError: No module named 'requests'"
# Node.js: "Error: Cannot find module 'express'"
# Go: "panic: runtime error: invalid memory address"
# Java: "Exception in thread main java.lang.NoClassDefFoundError"

Solutions:

  • Fix application code bugs (add error handling, fix imports)
  • Install missing dependencies in container image (pip install, npm install)
  • Validate configuration files before deployment
  • Add required command-line arguments to pod spec

Cause 2: Missing Environment Variables

Symptoms: Application expects environment variable but it's not set, crashes on startup with configuration error

Example error in logs:

Error: DATABASE_URL environment variable is required but not set
Fatal: Cannot start without DATABASE_URL

How to diagnose:

# Check what environment variables are actually set
kubectl exec payment-service-7d9f8b-xyz -- env
# (Won't work if pod keeps crashing immediately)

# Check pod spec to see what should be set
kubectl get pod payment-service-7d9f8b-xyz -o yaml | grep -A 10 env:

# Check if ConfigMap/Secret referenced exists
kubectl get configmap app-config
kubectl get secret app-secrets

Solutions:

# Add missing environment variable
kubectl set env deployment/payment-service DATABASE_URL=postgres://db:5432/mydb

# Or reference from ConfigMap
env:
- name: DATABASE_URL
  valueFrom:
    configMapKeyRef:
      name: app-config
      key: database_url

# Or from Secret
env:
- name: DB_PASSWORD
  valueFrom:
    secretKeyRef:
      name: db-secret
      key: password

Cause 3: Missing or Misconfigured ConfigMap or Secret

Symptoms: Pod events show CreateContainerConfigError, container never starts

Example:

kubectl describe pod my-pod

# Events show:
# Error: configmap "app-config" not found
# CreateContainerConfigError: secret "db-credentials" not found

Solutions:

# Verify ConfigMap exists in correct namespace
kubectl get configmap app-config -n production

# Create if missing
kubectl create configmap app-config \
  --from-literal=database_host=postgres.production.svc \
  --from-literal=log_level=info

# Verify Secret exists
kubectl get secret db-credentials -n production

# Create if missing
kubectl create secret generic db-credentials \
  --from-literal=username=myapp \
  --from-literal=password=SecurePass123

Cause 4: OOMKilled (Out of Memory)

Symptoms: Container crashes with exit code 137, describe shows "Reason: OOMKilled"

How to diagnose:

# Check termination reason and exit code
kubectl get pod my-pod -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'
# Output: OOMKilled

kubectl get pod my-pod -o jsonpath='{.status.containerStatuses[0].lastState.terminated.exitCode}'
# Output: 137 (means killed by SIGKILL, typically OOMKill)

# Check memory limit
kubectl get pod my-pod -o jsonpath='{.spec.containers[0].resources.limits.memory}'
# Output: 512Mi (might be too low)

# Check actual memory usage before crash (if metrics available)
kubectl top pod my-pod
# Or query Prometheus: container_memory_working_set_bytes{pod="my-pod"}

Solutions:

# Increase memory limits
kubectl set resources deployment/my-app \
  -c=my-container \
  --limits=memory=1Gi \
  --requests=memory=512Mi

# Or edit deployment directly
spec:
  containers:
  - name: app
    resources:
      requests:
        memory: 512Mi  # Guaranteed allocation
      limits:
        memory: 1Gi    # Maximum before OOMKill

Long-term solution: Investigate memory leaks in application code if memory usage grows continuously. Use profiling tools to identify leak sources.

Cause 5: Failed Liveness Probe

Symptoms: Container runs for a while then gets killed, events show "Liveness probe failed"

How probes cause CrashLoopBackOff:

  1. Container starts successfully
  2. Application initializing (loading data, connecting to database)
  3. Liveness probe executes before app ready (initialDelaySeconds too short)
  4. Probe fails (HTTP 404, command returns error, TCP connection refused)
  5. After failureThreshold consecutive failures, Kubernetes kills container
  6. Container restarts, same cycle repeats → CrashLoopBackOff

Check probe configuration:

kubectl get pod my-pod -o yaml | grep -A 10 livenessProbe

# Example problematic config:
livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 5   # Too short! App needs 30s to start
  periodSeconds: 10
  failureThreshold: 3      # Kills after 3 failures = 30s

Solutions:

# Increase initialDelaySeconds
livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 60  # Give app time to start
  periodSeconds: 10
  timeoutSeconds: 5
  failureThreshold: 3

# Or remove liveness probe temporarily to isolate issue
# (edit deployment, comment out livenessProbe section)

Causes 6–15: The Other Common Patterns

The remaining ten causes are diagnosed with the same three commands — kubectl describe pod, kubectl logs --previous and the exit code — so here is what to look for in each:

  • 6. Database connection failures: logs show "connection refused" or a timeout to the database. Check that the dependency's Service has endpoints, then add connection retries or an init container that waits for it.
  • 7. Wrong image or architecture: logs show "exec format error" when an amd64-only image lands on an arm64 node (or the reverse). Build a multi-arch image or constrain the pod to matching nodes.
  • 8. Permission denied: exit code 126 or "permission denied" in the logs. Check the pod's securityContext (runAsUser, fsGroup, read-only root filesystem) against the file modes inside the image.
  • 9. Missing runtime dependencies: "error while loading shared libraries" or a missing binary in a slim base image. Add the package to the image rather than installing it at startup.
  • 10. Port already in use: "address already in use" — usually two containers in the same pod binding the same port. Give each container its own port.
  • 11. File not found: the app cannot find a config or data file because volumeMounts.mountPath does not match the path the app reads. Align the mount path with the application.
  • 12. Incorrect command or entrypoint: exit code 127 or "executable file not found in $PATH". Fix command/args in the pod spec or the image's ENTRYPOINT.
  • 13. Wrong health check endpoint: the liveness probe targets a path or port the app does not serve, so a healthy container is killed. Events show "Liveness probe failed" with a 404 or connection refused; fix the probe path and port.
  • 14. Insufficient CPU: a tight CPU limit throttles startup until probes time out. Raise the CPU limit, or add a startupProbe so liveness checks wait for startup to finish.
  • 15. Network issues: DNS failures at startup such as "could not resolve host". Check CoreDNS health and any NetworkPolicy that blocks egress to DNS or the dependency.

Systematic Debugging Process for CrashLoopBackOff

Step-by-Step Investigation Methodology

Step 1: Identify the Failing Pod

# List pods with status
kubectl get pods | grep CrashLoopBackOff

Step 2: Check Pod Events

# View pod events (shows restart history, errors)
kubectl describe pod my-pod

# Focus on Events section at bottom:
# Events:
#   Type     Reason     Age                  From               Message
#   ----     ------     ----                 ----               -------
#   Normal   Scheduled  5m                   default-scheduler  Successfully assigned
#   Normal   Pulled     5m                   kubelet            Successfully pulled image
#   Normal   Created    5m                   kubelet            Created container
#   Normal   Started    5m                   kubelet            Started container
#   Warning  BackOff    2m (x10 over 4m)     kubelet            Back-off restarting failed container

Step 3: Get Container Exit Code

# Check why container terminated
kubectl get pod my-pod -o jsonpath='{.status.containerStatuses[0].lastState.terminated.exitCode}'
# Common codes:
# 0 = success (shouldn't cause CrashLoopBackOff)
# 1 = general error (check logs for specifics)
# 2 = misuse of shell command
# 126 = command cannot execute (permission issue)
# 127 = command not found (wrong path or binary missing)
# 137 = SIGKILL (often OOMKilled - check memory limits)
# 139 = SIGSEGV segmentation fault
# 143 = SIGTERM graceful shutdown

kubectl get pod my-pod -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'
# Output: Error, OOMKilled, Completed, etc.

Step 4: Examine Container Logs from Crashed Instance

# CRITICAL: Use --previous to get logs from crashed container
kubectl logs my-pod --previous

# Current container (kubectl logs my-pod without --previous) might have
# no useful logs if crashing immediately on startup

# Common errors to search for:
kubectl logs my-pod --previous | grep -i error
kubectl logs my-pod --previous | grep -i exception
kubectl logs my-pod --previous | grep -i fatal
kubectl logs my-pod --previous | grep -i panic

# Look for last messages before crash (usually at end of log)
kubectl logs my-pod --previous | tail -50

Step 5: Check Resource Limits and Usage

# View configured limits
kubectl get pod my-pod -o jsonpath='{.spec.containers[0].resources}'

# Check if OOMKilled
kubectl describe pod my-pod | grep -i oom

# If metrics-server installed, check resource usage
kubectl top pod my-pod

Step 6: Verify Dependencies

# Check if ConfigMaps and Secrets exist
kubectl get configmap -n production
kubectl get secret -n production

# Verify referenced resources
kubectl get pod my-pod -o yaml | grep -E "configMapKeyRef|secretKeyRef"

How Alert6 Automates CrashLoopBackOff Investigation

Traditional Manual Debugging (about 70 minutes)

Scenario: Payment service enters CrashLoopBackOff at 2:30 PM

  1. 2:30 PM - Pod crashes, engineer gets Slack alert
  2. 2:35 PM - Engineer opens terminal, runs kubectl get pods
  3. 2:37 PM - Identifies payment-service in CrashLoopBackOff
  4. 2:40 PM - Runs kubectl describe pod, reads through output
  5. 2:45 PM - Sees exit code 1 in events, not immediately helpful
  6. 2:48 PM - Runs kubectl logs --previous, gets 500 lines of logs
  7. 2:55 PM - Scrolls through logs searching for errors
  8. 3:00 PM - Finds "connection to postgres refused" error
  9. 3:05 PM - Checks if database pod is running
  10. 3:08 PM - Database running, checks database logs
  11. 3:15 PM - Realizes database restarted 2 minutes before payment service crash
  12. 3:20 PM - Queries Prometheus for database metrics
  13. 3:25 PM - Sees database was OOMKilled at 2:28 PM
  14. 3:30 PM - Correlates manually: DB OOMKill → restart → payment service connection refused → crash
  15. 3:32 PM - Fixes database memory limits
  16. 3:40 PM - Both services recover

Total time: 70 minutes of active debugging

AI-Assisted Investigation with Alert6

Same scenario with an AI SRE platform such as Alert6 (illustrative):

  1. The payment-service CrashLoopBackOff alert and the database OOMKill and restart alerts are correlated into one incident and routed to the on-call engineer once, instead of as separate pages
  2. The investigation checks the relevant telemetry, pod events, container logs and recent deployments or configuration changes
  3. It connects the evidence — database OOMKill → restart → connection refused → payment crash — and presents that as the probable root cause
  4. It proposes a remediation runbook with commands, a verification step and a confidence score, and posts the incident to Slack
  5. An authorized engineer approves it, or policy-governed automation runs it only if that runbook is explicitly allow-listed

Example incident summary (illustrative):

CrashLoopBackOff Detected: payment-service-7d9f8b-xyz

Status: Pod crashing continuously (5 restarts in 10 minutes)
Impact: Payment processing unavailable, ~8 transactions failed

ROOT CAUSE: PostgreSQL database pod was OOMKilled at 2:28 PM due to insufficient memory (used 510Mi of 512Mi limit). Database restarted and entered 2-minute recovery mode. Payment service attempted connection at 2:30 PM during recovery window and crashed with "connection refused" error.

TIMELINE:

  • 2:28:12 PM - Database memory usage: 510Mi / 512Mi limit (99.6%)
  • 2:28:14 PM - Database pod OOMKilled (exit code 137)
  • 2:28:20 PM - Database pod auto-restarted by Kubernetes
  • 2:28:25 PM - PostgreSQL starting recovery (replaying WAL logs)
  • 2:30:15 PM - Payment service attempted connection to database
  • 2:30:17 PM - Connection refused (DB still in recovery, not accepting connections)
  • 2:30:19 PM - Payment service crashed (exit code 1)
  • 2:30:25 PM - Alerts correlated into one incident and investigation started

FIX:

# Increase database memory to prevent future OOMKills
kubectl set resources statefulset/postgres \
  -c postgres \
  --limits=memory=1Gi \
  --requests=memory=768Mi

# Recovery: Database will stabilize in ~60 seconds
# Payment service will auto-recover once DB healthy

RECOMMENDATION: Implement connection retry logic in payment service with exponential backoff to handle transient database unavailability gracefully.

Engineer action (5 minutes):

  1. 2:35 PM - Engineer reads Slack RCA
  2. 2:36 PM - Approves the proposed runbook
  3. 2:37 PM - Verifies database and payment service recovered
  4. 2:40 PM - Creates Jira ticket for connection retry implementation

Total time: 10 minutes (detection to resolution) with only 5 minutes active engineering work

In this illustrative scenario, active engineering work drops from 70 minutes of evidence gathering to a few minutes of reviewing evidence and approving a fix.

Preventing CrashLoopBackOff: Best Practices

1. Test Thoroughly in Dev/Staging

Never deploy to production without testing in lower environments first.

2. Implement Proper Health Checks

Liveness probes should check actual application health, not just "is process running"

3. Set Appropriate Resource Limits

Monitor actual usage and set limits 20-30% above peak to handle spikes

4. Use Init Containers for Dependencies

If a container crashes because a dependency is not ready yet, add an init container that waits until the database or upstream Service answers, so the main container only starts when it can connect. Init containers run to completion before the app container starts.

5. Implement Graceful Degradation

Retry dependency connections with exponential backoff instead of exiting on the first failure, and use timeouts and circuit breakers so a slow or missing downstream degrades one feature rather than crashing the whole process.

Conclusion: Master CrashLoopBackOff Debugging

A CrashLoopBackOff error in Kubernetes is fixable once you identify the root cause. Follow systematic debugging: check exit code, examine logs with --previous, verify configuration, test dependencies, and adjust resources.

Key Takeaways:

  • Kubernetes CrashLoopBackOff = container crashes repeatedly, Kubernetes restarts with exponential backoff
  • Exit code 137 = OOMKilled (increase memory limits)
  • Exit code 1 = application error (check logs for specifics)
  • Always use kubectl logs --previous to see crashed container logs
  • Common causes: missing config, OOMKilled, failed probes, dependency unavailable
  • Systematic debugging beats random troubleshooting
  • AI SRE platforms such as Alert6 correlate the alerts, investigate the evidence and propose the fix for approval

Crash loops with exit code 137 are memory exhaustion wearing a different name, which is why the same investigation keeps recurring across issue types. AI SRE for Kubernetes covers running that investigation with an agent gathering evidence, and Alert6 is where it runs.

Want the evidence gathered before you open a terminal? Book an Alert6 demo and watch a crash loop correlated, investigated and handed back as a proposed fix your team approves.