The constraint was real: a single 512MB Render instance had to run a FastAPI server, a Celery worker, and a Celery beat scheduler; simultaneously, in production, without OOM failures on Render or job loss. No vertical scaling. No splitting services. Just make it work in the budget we had.
This was for Coopwise, a fintech platform I was building. The easy answer was to throw money at it — bump the instance size, separate the worker, orchestrate all services using Docker. But I wanted to understand the problem first, because if we don't understand why we were running out of memory, a bigger machine just delays the reckoning.
What was actually eating memory Celery's default prefork pool spawns multiple worker processes. Each one has its own Python interpreter, its own copy of imported modules, its own Redis connection pool. On a memory-constrained instance, this is death by a thousand forks. The workers weren't doing heavy work, they were just existing(it was a small project), and that costs more than the actual job execution.
The second problem: If the FastAPI server and both Celery processes all start at boot simultaneously, you get a spike that almost definitely kills the instance before any of them finish initializing.
What I built I wrote a Python supervisor, a single entry point that owns the full lifecycle of all three processes. Here's what it does:
Redis health-check before anything starts. The supervisor pings Redis and validates the connection before spawning a single worker. If Redis isn't up, nothing starts. This eliminated a whole class of failed startup cycles where workers would spin up, fail to connect, crash, and retry - consuming memory on every attempt.
Solo pool, concurrency 1, on Render. Switching from prefork to Celery's solo pool runs tasks in the same process as the worker. No forking, no child processes, no per-worker interpreter overhead. Combined with --max-memory-per-child=150000, this capped the blast radius of any leaky task.
Staggered startup. Worker -> 3s delay -> Beat -> 2s delay -> API. This flattens the initialization spike and ensures Redis is being hit by one process at a time on boot.
Gossip, mingle, and heartbeat disabled. These are Celery features that handle cluster coordination - useful when you have many workers talking to each other, overhead when you have one. Turning them off reduced idle CPU and memory chatter.
The supervisor also monitors all three processes in a loop and restarts any that crash. In development it fails fast; in production it recovers silently. The environment flag is just RENDER=true.
The Results The supervisor monitors all three processes in a loop and restarts any that crash. In development, it fails fast; in production, it recovers silently. The numbers speak for themselves: idle memory usage dropped by approximately 40%, and we maintained 97.9% uptime over 4 weeks on that same 512MB instance.
What this taught me The thing about Redis-backed job systems is that the failure modes aren't usually in the jobs themselves - they're in the bootstrap. A worker that can't connect to Redis on startup, a heartbeat loop that consumes memory while doing nothing useful, a pool that forks three processes when one will do. These are the things that kill you at 3am when no one's watching the dashboard.
Building this made me think differently about queue infrastructure in general. The questions that matter aren't just "is the job processed?" but "what happens when the broker is briefly unavailable?", "what's the per-idle-worker memory cost?", "what restarts what, and in what order?" Those questions are just as important in a Node.js/BullMQ context as they are in Python/Celery - the runtime changes, but the problem shape is the same.
This is the kind of problem I want to be solving at scale - which is part of why I'm currently exploring contributing to infrastructure-level open source work.