
How to Fix Subprocess Zombie Leaks and File Descriptor Exhaustion in AI Agents on Linux
Long-running autonomous coding agents run hundreds of terminal commands per hour. When agents spawn shell tools without proper process reaping, orphaned grandchild processes turn into <defunct> zombies. Eventually, the Linux system hits the PID limit or exhausts available file descriptors on unclosed IPC pipes, triggering BlockingIOError: [Errno 11] Resource temporarily unavailable and OSError: [Errno 24] Too many open files.
Here is how to configure a Linux child subreaper in Python, drain pipes without leaking file descriptors, and inspect process leaks in production containers.
1. Quick Fix: The Python Linux Subreaper Pattern
Mark your agent supervisor process as a child subreaper using prctl so that orphaned background processes re-parent to the agent rather than getting lost:
# Save as agent_subreaper.py
import ctypes
import os
import signal
import sys
PR_SET_CHILD_SUBREAPER = 36
def enable_subreaper():
# Declares the current process as an explicit subreaper for orphaned children
if sys.platform != "linux":
return
libc = ctypes.CDLL("libc.so.6")
res = libc.prctl(PR_SET_CHILD_SUBREAPER, 1, 0, 0, 0)
if res != 0:
raise OSError("Failed to set PR_SET_CHILD_SUBREAPER")
def reap_zombie_children(signum, frame):
# Reaps all exited background child processes without blocking
while True:
try:
pid, status = os.waitpid(-1, os.WNOHANG)
if pid <= 0:
break
except ChildProcessError:
break
# Register signal handler and enable subreaper
enable_subreaper()
signal.signal(signal.SIGCHLD, reap_zombie_children)
By setting PR_SET_CHILD_SUBREAPER, any intermediate child shell (such as a timed-out bash tool or subagent worker) that exits before its own children will pass those orphaned processes to your agent supervisor. The non-blocking SIGCHLD handler calls waitpid(-1, os.WNOHANG) to reap their exit codes instantly.
2. Root Cause Analysis: How Agents Leak PIDs and Pipes
AI agent loops fail through two distinct Linux kernel mechanisms:
| Leak Type | Root Cause | Trigger Event | Linux Kernel Error | Mitigation |
|---|---|---|---|---|
Zombie <defunct> |
Parent exits without waiting for child exit status | Tool timeout kills intermediate bash wrapper, leaving grandchild process | Errno 11 EAGAIN (PID exhaustion) |
PR_SET_CHILD_SUBREAPER + waitpid(WNOHANG) |
| Dangling Pipe FDs | subprocess.PIPE stdout/stderr buffers not explicitly closed |
Process terminates abnormally or read stream hangs on unread buffer | Errno 24 EMFILE (Too many open files) |
close_fds=True + explicit pipe context manager |
| Docker PID 1 Failure | Python runs as container PID 1 without init reaper | Docker default stops PID 1 from inheriting standard init signals | Process tree freezes on container restart | Use tini or docker run --init |
| IPC Buffer Deadlock | Command output exceeds 64KB Linux pipe buffer | Child blocks waiting for parent to read; parent blocks waiting for exit | Agent hanging at 100% timeout rate | communicate() or streaming async readers |
3. Safe Subprocess Execution with File Descriptor Cleanup
Never use raw subprocess.Popen without ensuring stdout and stderr pipes close under every exception path. Use this hardened execution wrapper:
import asyncio
import os
import signal
import subprocess
async def execute_agent_command(command: str, timeout_seconds: int = 60) -> tuple[int, str, str]:
# Executes a terminal command with guaranteed pipe cleanup and process group termination
proc = await asyncio.create_subprocess_shell(
command,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
preexec_fn=os.setsid, # Create independent process group
close_fds=True,
)
try:
stdout_bytes, stderr_bytes = await asyncio.wait_for(
proc.communicate(),
timeout=timeout_seconds,
)
return (
proc.returncode,
stdout_bytes.decode("utf-8", errors="replace"),
stderr_bytes.decode("utf-8", errors="replace"),
)
except asyncio.TimeoutError:
# Kill the entire process group, not just the parent shell
try:
os.killpg(os.getpgid(proc.pid), signal.SIGKILL)
except ProcessLookupError:
pass
# Wait for termination to clean up file descriptors
await proc.wait()
raise TimeoutError(f"Command timed out after {timeout_seconds}s: {command}")
Setting preexec_fn=os.setsid attaches all child processes spawned by the shell to a unique process group ID. When a timeout occurs, os.killpg(os.getpgid(proc.pid), signal.SIGKILL) kills all grandchildren simultaneously, preventing orphaned background commands from lingering.
4. Docker Container Configuration: Avoiding the PID 1 Trap
If you package your agent inside Docker, Python running as PID 1 will not reap orphaned processes by default.
Always run containers with the --init flag or incorporate tini:
# Hardened Agent Dockerfile
FROM python:3.12-slim-bookworm
# Install tini as init system
RUN apt-get update && apt-get install -y --no-install-recommends tini git \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
# Use tini as the entrypoint to handle PID 1 signal forwarding and child reaping
ENTRYPOINT ["/usr/bin/tini", "--"]
CMD ["python3", "main.py"]
In docker-compose.yml, specify init: true:
services:
agent-runner:
build: .
init: true
ulimits:
nofile:
soft: 65535
hard: 65535
nproc: 4096
restart: on-failure
5. Live Auditing: Inspecting Zombie Processes and File Descriptors
Verify whether your running agent process is leaking resources using standard Linux diagnostic tools:
# 1. Check for zombie defunct processes
ps aux | grep "[d]efunct"
# 2. Count open file descriptors for the agent PID
AGENT_PID=$(pgrep -f "main.py")
ls -l /proc/$AGENT_PID/fd | wc -l
# 3. Inspect open pipes and network sockets
lsof -p $AGENT_PID | grep -E "FIFO|PIPE|sock"
# 4. Check system-wide file handle allocation
cat /proc/sys/fs/file-nr
A stable agent maintaining 10 concurrent tasks should hold between 25 and 45 open file descriptors. If ls -l /proc/$AGENT_PID/fd | wc -l continuously climbs after each completed turn, inspect your tool handlers for unclosed streaming pipes or unawaited subprocesses.