Introduction to OOMKilled: When Memory Limits Are Exceeded

OOMKilled (Out Of Memory Killed) is one of the most disruptive and difficult-to-diagnose errors in Kubernetes, occurring when a container exceeds its configured memory limit and the Linux kernel's Out Of Memory (OOM) killer forcefully terminates the process to protect the node from complete memory exhaustion that could crash the entire system. Unlike graceful application shutdowns where your code can clean up resources, save state, and exit cleanly, OOMKilled is violent and immediate the kernel sends SIGKILL (signal 9) which cannot be caught, handled, or ignored by the application, instantly terminating the process mid-execution potentially corrupting data, losing in-flight transactions, breaking database connections without cleanup, and leaving the application in inconsistent state requiring recovery procedures when it restarts.

The OOMKilled problem manifests in several frustrating ways that impact production services: containers repeatedly crash and enter CrashLoopBackOff as Kubernetes restarts them, they hit the memory limit again, get OOMKilled again, creating an infinite loop of failures, deployments fail to roll out because new pods immediately get OOMKilled while trying to initialize, applications experience periodic crashes under load when memory usage spikes above limits during traffic bursts or batch processing jobs, databases lose connections and corrupt indexes when killed mid-transaction without proper shutdown, stateful applications lose cached data and warm-up state requiring minutes to rebuild after restart degrading performance, and customer-facing services become unreliable with random crashes that users experience as 500 errors, timeouts, or complete unavailability.

Understanding OOMKilled requires understanding how Linux memory management and Kubernetes resource limits interact: containers run in Linux cgroups (control groups) that enforce resource limits including memory, when a container's memory usage (specifically the "working set" which excludes reclaimable page cache) reaches the limit configured in resources.limits.memory in the pod specification, the cgroup memory controller triggers, Linux kernel's OOM killer selects a process to terminate based on OOM score (memory usage, OOM adjustment values), the selected process (usually your application's main process with highest memory consumption) receives SIGKILL terminating immediately, the container exits with exit code 137 (128 + 9 where 9 is the SIGKILL signal number), and Kubernetes sees the terminated container with reason "OOMKilled" and restarts it according to the restart policy, typically leading to CrashLoopBackOff if the problem persists.

This comprehensive guide teaches you everything about OOMKilled errors from root causes to permanent solutions, covering: what OOMKilled means technically and how to confirm it happened versus other container termination reasons, understanding Kubernetes memory metrics (working set bytes, RSS, cache, swap) and which one triggers OOMKill, the difference between memory requests and limits and how each affects scheduling versus runtime behavior, the most common root causes of OOMKilled including insufficient limits, memory leaks, configuration errors, and unexpected load, systematic debugging methodology to identify whether you need bigger limits or to fix a memory leak, analyzing memory usage patterns over time using Prometheus queries, calculating optimal memory requests and limits based on actual usage plus appropriate headroom for spikes, fixing memory leaks in applications through profiling and code analysis, and how an AI SRE platform such as Alert6 automates the investigation you'd otherwise do by hand.

By mastering OOMKilled troubleshooting through this guide, you'll be able to diagnose memory issues in minutes instead of hours, distinguish between legitimate need for more memory versus fixable memory leaks, optimize memory allocation balancing cost against reliability, prevent OOMKills through proper limit configuration and application tuning, and leverage AI-powered analysis to automate the entire investigation process.

What is OOMKilled? Technical Deep Dive into Linux OOM Killer

How the Linux OOM Killer Works

The Out Of Memory killer is a Linux kernel mechanism protecting system stability when memory is completely exhausted. When a container in Kubernetes exceeds its memory limit, here's exactly what happens:

  1. Memory Usage Grows: Your application allocates more memory (malloc in C, new objects in Java/Python/Node.js, slice/map growth in Go)
  2. Approaching Limit: Container memory working set approaches the limit configured in resources.limits.memory (e.g., 512Mi)
  3. Limit Exceeded: Working set reaches or exceeds limit (512Mi)
  4. Cgroup Controller Triggers: Linux cgroup memory controller detects limit violation
  5. OOM Killer Invoked: Kernel's OOM killer mechanism activates
  6. Process Selection: OOM killer selects victim process based on OOM score (highest memory user typically selected)
  7. SIGKILL Sent: Kernel sends SIGKILL (signal 9 - cannot be caught or blocked) to victim process
  8. Process Terminated: Application terminates immediately without cleanup, exit code 137 (128 + 9)
  9. Container Exits: Container stops with reason "OOMKilled"
  10. Kubernetes Restart: Restart policy (usually Always) triggers automatic restart

Why Exit Code 137? Unix convention: 128 + signal number. SIGKILL is signal 9, so 128 + 9 = 137. This exit code specifically indicates forceful termination by signal, and in Kubernetes context with "OOMKilled" reason almost always means memory limit exceeded.

Confirming OOMKilled Happened

# Method 1: Check termination reason
kubectl get pod my-pod -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'
# Output: OOMKilled (confirms it)

# Method 2: Check exit code
kubectl get pod my-pod -o jsonpath='{.status.containerStatuses[0].lastState.terminated.exitCode}'
# Output: 137 (very likely OOMKilled, especially with reason above)

# Method 3: Describe pod and look for OOMKilled in events
kubectl describe pod my-pod

# Events section shows:
# State:          Terminated
#   Reason:       OOMKilled
#   Exit Code:    137
#   Started:      Mon, 27 Oct 2025 14:30:15 +0000
#   Finished:     Mon, 27 Oct 2025 14:32:47 +0000

Understanding Kubernetes Memory Metrics

Critical memory metrics explained:

  • container_memory_working_set_bytes: THIS is what counts toward OOMKill limit. Working set = RSS (anonymous memory) + page cache that cannot be reclaimed. When this equals or exceeds limits.memory, OOMKill triggers.
  • container_memory_rss: Resident Set Size - anonymous memory (heap, stack, not backed by files). Always less than or equal to working set.
  • container_memory_cache: Page cache memory used for filesystem caching. Mostly reclaimable but some counted in working set if actively used.
  • container_memory_usage_bytes: Total memory including all caches. Higher than working set, NOT what triggers OOMKill (misleading metric!).

The confusion: usage vs working set

Many engineers check container_memory_usage_bytes, see 980Mi against a 1Gi limit and panic, when most of that is reclaimable page cache and container_memory_working_set_bytes is only 600Mi. Because the working set is always less than or equal to usage, usage over-states OOMKill risk; the working set is the number the kernel acts on. Always monitor working_set, not usage.

The 8 Most Common Root Causes of OOMKilled

Cause 1: Memory Limit Set Too Low for Normal Operation

Symptoms: Application crashes during normal operation, memory usage steady at limit before crash, happens predictably under certain operations

How to diagnose:

# Check current memory limit
kubectl get pod my-pod -o jsonpath='{.spec.containers[0].resources.limits.memory}'
# Output: 512Mi

# Check memory usage before crash (if Prometheus available)
# Query for last 24 hours before OOMKill:
container_memory_working_set_bytes{pod="my-pod"}

# Typical pattern: Usage steadily at 500-512Mi (98-100% of limit)
# Indicates limit too low, not a leak

Solution:

# Increase memory limit appropriately
kubectl set resources deployment/my-app \
  -c=my-container \
  --limits=memory=1Gi \
  --requests=memory=768Mi

# Rule of thumb: Set limit 20-30% above typical p95 usage
# If p95 usage is 750Mi, set limit to 1Gi (33% headroom)

Example analysis (illustrative):

OOMKilled Analysis: database-primary-0

Current Config:

  • Memory Limit: 512Mi
  • Memory Request: 256Mi

Usage Analysis (30-day history):

  • Average usage: 450Mi (88% of limit)
  • P95 usage: 505Mi (99% of limit)
  • P99 usage: 510Mi (99.6% of limit)
  • OOMKills: 12 times in last 30 days
  • Pattern: No leak detected (usage plateaus at ~500Mi)

Root Cause: Memory limit too low for normal database operation. During active query processing, memory naturally reaches 500-510Mi hitting limit.

Recommended Fix:

kubectl set resources statefulset/database \
  -c postgres \
  --limits=memory=768Mi \
  --requests=memory=512Mi

Rationale: P99 usage (510Mi) + 30% safety margin = 663Mi, rounded to 768Mi
Cost Impact: +$12/month per pod
Expected OOMKills after fix: 0 (adequate headroom)

Stop running the OOMKill investigation by hand.

Alert6 correlates the OOMKill with the alerts it triggered, investigates the evidence and proposes a fix your team approves.

Book a demo →

Cause 2: Memory Leak in Application Code

Symptoms: Memory usage grows continuously over time, never plateaus, eventually hits limit, crashes, restarts, grows again, crashes again

How to diagnose:

# Query Prometheus for memory growth over time
container_memory_working_set_bytes{pod=~"my-app-.*"}

# Pattern indicating leak:
# Hour 0: 200Mi
# Hour 6: 350Mi  
# Hour 12: 500Mi
# Hour 15: 512Mi -> OOMKill -> restart
# Hour 16: 200Mi (after restart)
# Hour 22: 350Mi (growing again)
# Continuous linear growth = memory leak

# Calculate growth rate
rate(container_memory_working_set_bytes[1h])
# If consistently positive and never negative, probable leak

Solutions:

Short-term (band-aid):

# Increase limit to buy time
kubectl set resources deployment/my-app --limits=memory=2Gi

# This delays OOMKill but doesn't fix leak

Long-term (actual fix):

  • Profile application memory: Use language-specific profilers (pprof for Go, heapdump for Java, memory_profiler for Python, heapdump for Node.js)
  • Identify leak source: Find objects accumulating without being freed (caches never cleaned, event listeners never removed, connections not closed)
  • Fix code: Add proper cleanup, implement cache eviction, close resources, remove listeners
  • Test: Run load tests monitoring memory over time, verify memory plateaus

Common leak patterns by language:

  • Go: Goroutine leaks, slice/map growth without bounds, defer in loops
  • Java: Static collections, ThreadLocal not cleaned, listeners not removed
  • Python: Global caches, circular references preventing GC, unclosed file handles
  • Node.js: Event listeners, unclosed connections, global array accumulation

Example leak analysis (illustrative):

Memory Leak Detected: api-server-deployment

Usage Pattern Analysis (7-day history):

  • Memory growth rate: +18Mi per hour (continuous linear increase)
  • Starting memory (after restart): 280Mi
  • Projected time to OOMKill: 11 hours from restart
  • OOMKill frequency: Every 10-12 hours consistently
  • Leak confidence: 95% (high certainty based on pattern)

Leak Signature: Memory never plateaus, growth never stops, restart temporarily fixes (drops to 280Mi) but growth resumes immediately

Immediate Action: Increase limit to prevent crashes while leak is fixed

kubectl set resources deployment/api-server --limits=memory=2Gi

Permanent Fix Required: Profile application, identify leak source, fix code
Estimated leak location: Based on logs, likely unclosed database connections accumulating

Cost of band-aid: +$45/month (doubled memory)
Cost of not fixing: Continued crashes every 11 hours, customer impact, engineering time

Causes 3-8: The Other Patterns Worth Ruling Out

Cause 3 — Sudden memory spike under load: Usage is healthy at steady state but a traffic burst, cache warm-up, or batch job pushes the working set past the limit in minutes. Diagnose by correlating OOMKill timestamps with request-rate or job-schedule metrics. Fix with limits sized to p99-under-load, HPA scaling before saturation, or moving batch work to dedicated pods with their own limits.

Cause 4 — Too many concurrent requests: Each in-flight request holds buffers and objects; unbounded concurrency means memory scales with traffic. Fix with connection/worker pool caps, bounded queues, and backpressure (return 503 rather than OOM).

Cause 5 — Large file or payload processing: Loading whole files or result sets into memory (instead of streaming) spikes usage proportional to input size. Fix by streaming, chunking, and enforcing payload size limits at ingress.

Cause 6 — Inefficient or unbounded caching: In-process caches without eviction grow until the limit. Fix with size-bounded caches (LRU), TTLs, or moving the cache out of process (Redis/Memcached).

Cause 7 — JVM (or runtime) heap misconfiguration: A JVM with -Xmx equal to (or above) the container limit leaves no room for non-heap memory — metaspace, threads, direct buffers — so the container OOMs even though the heap looks fine. Rule of thumb: max heap at ~75% of the container limit. The Node.js equivalent is --max-old-space-size.

Cause 8 — Sidecars and shared pod memory: The pod limit covers every container in it. A logging or mesh sidecar quietly consuming 200Mi shrinks the headroom of the main container. Diagnose per-container (not per-pod) working set; give each container its own explicit requests and limits.

Systematic OOMKilled Debugging Process

Step 1: Confirm OOMKilled (Not Other Crash Reason)

# Check all three indicators
kubectl get pod my-pod -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'
# Should show: OOMKilled

kubectl get pod my-pod -o jsonpath='{.status.containerStatuses[0].lastState.terminated.exitCode}'  
# Should show: 137

kubectl describe pod my-pod | grep -A 5 "Last State"
# Shows: Reason: OOMKilled, Exit Code: 137

Step 2: Check Current Memory Limit

# What limit is configured?
kubectl get pod my-pod -o jsonpath='{.spec.containers[0].resources.limits.memory}'
# Example: 512Mi

# Also check request
kubectl get pod my-pod -o jsonpath='{.spec.containers[0].resources.requests.memory}'
# Example: 256Mi

Step 3: Analyze Historical Memory Usage

# If Prometheus available, query memory trends
# Last 24 hours before OOMKill
container_memory_working_set_bytes{pod="my-pod"}[24h]

# Calculate p95 and p99 usage
quantile_over_time(0.95, container_memory_working_set_bytes{pod=~"my-app-.*"}[7d])
quantile_over_time(0.99, container_memory_working_set_bytes{pod=~"my-app-.*"}[7d])

# If metrics-server installed (real-time only, no history)
kubectl top pod my-pod

Step 4: Determine If Memory Leak or Insufficient Limit

Memory Leak Indicators:

  • Memory usage grows continuously over time
  • Never plateaus or reaches steady state
  • Growth rate is consistent (e.g., +15Mi per hour)
  • After restart, starts low then grows again
  • Pattern repeats predictably

Insufficient Limit Indicators:

  • Memory usage plateaus at or near limit
  • Usage stable at 480-500Mi when limit is 512Mi
  • OOMKills happen during known high-memory operations (batch jobs, cache warming)
  • No continuous growth usage is stable until specific events

Step 5: Calculate Optimal Limits

Statistical approach:

# Get p95 usage over last 30 days
p95_usage = quantile_over_time(0.95, container_memory_working_set_bytes[30d])

# Calculate recommended limit
recommended_limit = p95_usage * 1.3  # 30% headroom for spikes

# Example:
# p95 usage = 650Mi
# recommended limit = 650Mi * 1.3 = 845Mi
# Round up to 1Gi for cleaner configuration

Rule of thumb:

  • Stateless apps: p95 + 20% (smaller spikes expected)
  • Stateful apps (databases, caches): p99 + 30% (larger variance)
  • Batch jobs: max observed + 50% (highly variable)

You Just Did All of This by Hand. Here Is the Automated Version.

Notice what the five steps above actually are: pull termination state, pull the configured limits, query Prometheus history, classify leak-vs-limit from the growth curve, compute a p95/p99-based target. That's a root-cause analysis — and every time a pod OOMKills at 2am, someone on your team runs this loop manually.

An AI SRE platform such as Alert6 runs this kind of investigation automatically. It correlates the related alerts into one incident (the OOMKill, the CrashLoopBackOff it caused, the latency alerts that fired), investigates live telemetry, logs and recent deployments or configuration changes, and presents a probable root cause with the evidence behind it. The fix arrives as a remediation runbook with commands, verification steps and a confidence score; an authorized engineer approves it, or policy-governed automation runs it only if that runbook is explicitly allow-listed, and every approval and action is recorded. Whoever applies it, land the limit change in Git: a kubectl patch someone runs at 2am is drift waiting to be reverted by the next sync — a merged PR is a fix that survives.

The limit itself is a sizing question. If OOMKills keep tracing back to requests and limits that were set once and never revisited, SpendZero compares each container's CPU and memory requests with its 14-day p95 usage and recommends new values with headroom.

Fixing OOMKilled: Immediate and Long-Term Solutions

Immediate Fix: Increase Memory Limits

# Quick fix to stop crashes
kubectl set resources deployment/my-app \
  --limits=memory=1Gi \
  --requests=memory=768Mi

# Verify change applied
kubectl rollout status deployment/my-app

# Monitor to confirm OOMKills stop
kubectl get pods --watch

When to increase limits:

  • Memory usage consistently at 90-100% of limit
  • No evidence of memory leak (usage plateaus)
  • Application legitimately needs more memory for its workload

Long-Term Fix: Find and Fix Memory Leaks

Memory Profiling by Language:

Go Applications:

# Enable pprof in application
import _ "net/http/pprof"

# Expose on port 6060
http.ListenAndServe(":6060", nil)

# Collect heap profile
kubectl port-forward my-pod 6060:6060
go tool pprof http://localhost:6060/debug/pprof/heap

# Analyze top memory consumers
(pprof) top10
(pprof) list

Java Applications:

# Take heap dump when memory high
kubectl exec my-pod -- jmap -dump:live,format=b,file=/tmp/heap.bin 1

# Copy heap dump locally
kubectl cp my-pod:/tmp/heap.bin ./heap.bin

# Analyze with Eclipse MAT or VisualVM
# Look for: Objects with high retained size, memory leak suspects, duplicate strings

Python Applications:

# Use memory_profiler
pip install memory-profiler

# Decorate functions
@profile
def my_function():
    # function code

# Run with profiler
python -m memory_profiler app.py

# Or use tracemalloc (built-in Python 3.4+)
import tracemalloc
tracemalloc.start()
# ... run code ...
snapshot = tracemalloc.take_snapshot()
top_stats = snapshot.statistics('lineno')
for stat in top_stats[:10]:
    print(stat)

Node.js Applications:

# Take heap snapshot
kubectl exec my-pod -- node --expose-gc --inspect=0.0.0.0:9229 app.js

# Use Chrome DevTools or clinic.js for analysis
npm install -g clinic
clinic doctor -- node app.js

Memory Optimization Best Practices

1. Set Memory Requests and Limits Appropriately

resources:
  requests:
    memory: 512Mi  # Guaranteed allocation (for scheduling)
  limits:
    memory: 768Mi  # Maximum before OOMKill (50% headroom)

Best practices:

  • Request = typical steady-state usage
  • Limit = p95 or p99 usage + 20-30% safety margin
  • Limit should be 1.5-2x request (not 10x)
  • Set the request explicitly: if you set only a limit, Kubernetes sets the request equal to the limit, which reserves more than the pod typically needs

2. Monitor Memory Usage Over Time

# Prometheus alert for high memory usage (warning before OOMKill)
- alert: HighMemoryUsage
  expr: |
    (container_memory_working_set_bytes
    /
    container_spec_memory_limit_bytes)
    > 0.9
  for: 10m
  annotations:
    summary: "Pod {{ $labels.pod }} using {{ $value | humanizePercentage }} of memory limit"

3. Implement Graceful Degradation

Instead of crashing when memory full, applications should:

  • Clear caches when memory pressure high
  • Reject new requests returning 503 Service Unavailable
  • Implement circuit breakers
  • Use bounded queues and buffers (not unlimited)

4. Use Horizontal Pod Autoscaling

# Scale pods based on memory usage
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 70  # Scale when avg memory > 70%

Distributes load across more pods, keeping individual pod memory usage lower. For the full scaling picture, see our HPA, VPA and Cluster Autoscaler guide.

5. Implement Memory Limits in Application Code

Java JVM:

# Set max heap based on container limit
# Container limit: 1Gi
# JVM max heap: 768Mi (75% of limit, leave room for non-heap memory)

JAVA_OPTS="-Xmx768m -Xms512m"

Node.js:

# Set V8 max old space
# Container limit: 1Gi = 1024Mi
# V8 limit: ~768Mi

node --max-old-space-size=768 app.js

How to Prevent and Diagnose OOMKilled Automatically

Proactive Memory Monitoring

Alert on memory before the kernel acts for you:

  • Warning at 85%: "Pod approaching memory limit (85%), OOMKill risk in 2-4 hours at current growth rate"
  • Critical at 95%: "Pod at 95% memory limit, OOMKill imminent within minutes"
  • Leak Detection: "Memory growing +12Mi/hour, probable leak, will OOMKill in 8 hours"

Post-OOMKill Analysis

When an OOMKill happens, an automated investigation should:

  1. Retrieve 24h memory usage history from Prometheus
  2. Calculate statistical patterns (mean, p50, p95, p99, growth rate)
  3. Determine if leak (continuous growth) or spike (sudden increase)
  4. Calculate the right limit based on usage + appropriate headroom
  5. Show the cost impact of the recommended limit increase
  6. Propose the manifest change for a human to review and merge
  7. If a leak is detected, suggest profiling with language-specific commands

Alert6 covers the incident side of that list: it correlates the alerts, investigates telemetry, logs and recent changes, and proposes a fix with a confidence score for an engineer to approve. Sizing the limit from long-run usage is a rightsizing job, which is what SpendZero's Kubernetes workload rightsizing does across every container, not just the one that crashed.

Cost-Aware Recommendations

Every limit change has a price, so put it next to the recommendation:

  • "Increasing from 512Mi to 1Gi adds $18/month per pod"
  • "Current pod has 5 replicas = $90/month total cost increase"
  • "Alternative: Reduce replicas from 5 to 4 and increase memory = net $0 cost change"

That lets you balance reliability and cost deliberately rather than by trial and error.

Conclusion: Master OOMKilled Prevention and Resolution

OOMKilled means container exceeded memory limit and was forcefully terminated. Fix by: confirming OOMKilled (exit code 137, reason OOMKilled), analyzing memory usage patterns (leak vs spike vs insufficient limit), increasing limits appropriately (p95/p99 + 20-30%), fixing memory leaks if detected through profiling, and monitoring proactively to prevent future OOMKills.

Key Takeaways:

  • OOMKilled = memory limit exceeded, kernel kills process
  • Exit code 137 = SIGKILL, usually indicates OOMKilled
  • Monitor container_memory_working_set_bytes (what triggers OOMKill)
  • Set limits = p95 usage + 30% headroom
  • Memory leak = continuous growth never plateauing
  • Profile and fix leaks, don't just increase limits indefinitely
  • Automate the investigation: an AI SRE platform such as Alert6 correlates the alerts, finds the probable cause and proposes the fix for approval
  • Balance cost vs reliability when sizing memory limits

If OOMKills are recurring rather than one-off, the sizing is a symptom and the loop around it is the problem. AI SRE for Kubernetes walks the same investigation with an agent gathering the evidence, and Alert6 is where that loop runs.

Tired of running this investigation by hand every time a pod dies? Book an Alert6 demo and watch an OOMKill correlated, investigated and handed back as a proposed fix your team approves. To find the workloads whose memory requests no longer match real usage, start a free SpendZero scan — read-only, no agents.