Inspecting Linux Kernel OOM Kills and Container Memory Limits

Docker Exit Code 137 indicates that the Linux kernel Out-Of-Memory (OOM) killer forcefully terminated your container via SIGKILL (128 + 9 = 137).

To stop OOM terminations immediately without upgrading server RAM, allocate a 4GB persistent swap file:

# 1. Create and activate a 4GB swap file on Linux
sudo fallocate -l 4G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

# 2. Verify swap activation
free -h

Next, configure explicit memory and swap ceilings in your docker run command per the Docker Engine resource constraints documentation:

docker run -m 2g --memory-swap 4g -d your-image-name

For AI model inference or deep learning containers running PyTorch, default 64MB shared memory limits cause hidden crashes. Always mount shared memory when launching GPU workloads:

docker run --gpus all --ipc=host --shm-size=16g -d your-ai-container

---

1. How to Confirm Exit Code 137

Before refactoring application code, verify whether the container stopped due to kernel OOM or an intentional manual termination:

docker inspect <container_id_or_name> --format='{{json .State}}' | jq .

Inspect these two JSON keys:

• "OOMKilled": true: The Linux kernel terminated the process because host or cgroup memory was exhausted.
• "ExitCode": 137: The container exited due to SIGKILL (signal 9).

You can inspect the host kernel ring buffer for exact process death logs:

sudo dmesg -T | grep -i -E 'oom[-_]killer|killed process|Out of memory'

Typical kernel output reveals which process exceeded its quota:

[Wed Oct  9 14:02:11 2026] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=docker-...,mems_allowed=0,oom_memcg=/docker/...,task_memcg=/docker/...,task=python3,pid=84920,uid=0
[Wed Oct  9 14:02:11 2026] Memory cgroup out of memory: Killed process 84920 (python3) total-vm:8542104kB, anon-rss:2048512kB, file-rss:128kB, shmem-rss:0kB

---

2. Inspecting Cgroup v2 Memory Metrics

Modern Linux distributions like Ubuntu 24.04 use Linux kernel Control Group v2 (cgroup v2). You can inspect exact memory consumption directly from the virtual filesystem:

# Locate container cgroup path
CONTAINER_ID=$(docker inspect --format='{{.Id}}' <container_name>)
CGROUP_PATH="/sys/fs/cgroup/system.slice/docker-${CONTAINER_ID}.scope"

# Read current memory usage and maximum limit
cat ${CGROUP_PATH}/memory.current
cat ${CGROUP_PATH}/memory.max
cat ${CGROUP_PATH}/memory.events

The memory.events file records an exact counter of OOM terminations:

• oom: Number of times the cgroup entered an out-of-memory state.
• oom_kill: Number of times processes in this cgroup were killed.

---

3. Four Resolution Strategies

Strategy A: Configure Docker Compose Memory Limits

In production multi-container architectures, set hard limits and memory reservations in docker-compose.yml:

services:
  web-api:
    image: my-service:latest
    deploy:
      resources:
        limits:
          memory: 4096M
        reservations:
          memory: 2048M
    restart: unless-stopped

If memory limits are omitted, a single memory leak can consume all host RAM, destabilizing sibling containers and causing host crashes.

Strategy B: Fix PyTorch and AI Workload Shared Memory

AI containers running local vLLM inference instances or multi-GPU training jobs frequently crash with exit code 137 despite having available VRAM.

Docker defaults to a 64MB /dev/shm shared memory mount. When PyTorch DataLoader multiprocessing workers exchange tensor batches across CPU processes, this 64MB buffer fills in seconds, triggering a DataLoader worker bus error crash.

Fix this by setting explicit shared memory in your run parameters:

# Set shared memory size to 16GB
docker run --shm-size=16g your-pytorch-image:latest

# Or share the host IPC namespace directly
docker run --ipc=host your-pytorch-image:latest

For GPU allocator tuning, check our guide on resolving PyTorch cuBLAS allocation failed errors.

Strategy C: Tune Application Runtime Heap Caps

Container runtimes like Node.js and Java do not always respect Docker memory boundaries automatically.

• Node.js: By default, V8 sets its heap limit based on total system RAM rather than container cgroup limits. Cap it to 75% of your container limit:

  # Inside container entrypoint
  NODE_OPTIONS="--max-old-space-size=1536" node server.js

• Java (JVM): Use modern container-aware percentage flags:

  java -XX:+UseContainerSupport -XX:MaxRAMPercentage=75.0 -jar application.jar

• Python: If executing scripts inside AI code execution sandboxes, set soft memory limits in Python using the resource module before processing untrusted inputs:

  import resource
  # Limit memory to 2GB
  max_bytes = 2 * 1024 * 1024 * 1024
  resource.setrlimit(resource.RLIMIT_AS, (max_bytes, max_bytes))

Strategy D: Adjust Linux Kernel OOM Badness Score

The Linux kernel selects which process to kill based on its oom_score. If your container runs a critical database (PostgreSQL, Redis) alongside background workers, protect the database by reducing its OOM score adjustment:

# Prevent database container from being killed first (-1000 to 1000)
docker run --oom-score-adj=-500 -d redis:alpine

---

Mitigation Methods Comparison

Solution Method	Implementation Time	Cost Impact	Production Recommendation
Adding Host Swap	Under 2 minutes	$0 (Disk space)	Essential for build servers & bursty workers
Enabling --shm-size=16g	Under 1 minute	$0	Mandatory for PyTorch, vLLM, and CUDA jobs
Runtime Heap Tuning	5 minutes	$0	Standard best practice for Node.js and Java
Vertical Hardware Scaling	10 minutes	+$15 to $60/month	Required for high-throughput enterprise APIs

---

Summary Checklist for Preventing Exit Code 137

1. Verify Root Cause: Run docker inspect <container> | grep OOMKilled to distinguish memory exhaustion from external SIGKILL commands.
2. Mount Shared Memory for AI: Always add --ipc=host or --shm-size=16g when serving deep learning containers or local Ollama instances.
3. Configure Persistent Swap: Ensure Linux VMs have at least 4GB of swap space enabled to absorb temporary memory spikes.
4. Cap Application Memory: Keep runtime heaps below 75% of container allocation limits.
5. Inspect Cgroup Events: Monitor /sys/fs/cgroup/.../memory.events to catch OOM conditions before production outages happen.

More in PyTorch & CUDA

PyTorch 2.15 Upgrade Guide: Python 3.11 Migration, CUDA Architecture Drops, and Linux Autograd Fixes

Troubleshoot PyTorch 2.15 breaking changes on Linux: fix Python 3.10 wheel deprecation, resolve no kernel image available on Volta, and repair autograd hooks.

How to Fix 'RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn' in PyTorch on Linux

Fix PyTorch RuntimeError: element 0 of tensors does not require grad on Linux. Repair broken autograd graphs, unfreeze layers, and remove no_grad blocks.

How to Fix 'RuntimeError: mat1 and mat2 shapes cannot be multiplied' in PyTorch on Linux

Resolve PyTorch RuntimeError: mat1 and mat2 shapes cannot be multiplied on Linux. Fix nn.Linear in_features mismatches, dynamic batch flattening, and attention head dimensions.