We run a job processing pipeline that handles 10,000+ tasks daily. One Monday morning, none of our high-priority jobs had processed for 18 hours. The queue had 1,275 low-priority bulk tasks blocking everything.
Here’s how we fixed it with Bull queue isolation - and why a single queue is a ticking time bomb.
The Problem
// DON'T DO THIS - single queue for everything
const queue = new Bull('jobs');
queue.process(3, async (job) => {
switch(job.data.type) {
case 'critical-import': return handleImport(job);
case 'bulk-import': return handleBulkImport(job); // Takes 30 min each!
}
});
At 3 AM, our scheduler enqueued 1,275 bulk import jobs. Each takes 5-30 minutes. With 3 workers, that’s days of processing. Meanwhile, our critical API imports (which take 2 minutes each) sat waiting.
The Fix: Dedicated Queues
class QueueManager {
constructor() {
// Critical path - never blocked
this.createQueue('import', { concurrency: 3 });
// Bulk operations - isolated
this.createQueue('import-bulk', { concurrency: 2 });
// Rate-limited sources - separate
this.createQueue('rate-limited', { concurrency: 2 });
}
}
Results:
- Critical imports: 0 delays since the change
- Bulk operations: Process at their own pace without affecting anything
- Rate-limited sources: Their 429s don’t block other queues
The Rule
If a job type can starve other job types, it needs its own queue. Period.
This pattern runs in production at MisuJob, maintaining 1M+ job listings from 50+ sources.
Have you been bitten by queue starvation? Share your story below.
