We have 1,000,000+ pages. Google’s Indexing API lets us notify Google about new and removed pages. But with a 200 requests/day quota and 433,000 pending URLs, we’d need 6 years to clear the queue.
Here’s how we architected around this limitation.
The Three Consumers Problem
Our system has three services that all consume the same 200/day Indexing API quota:
1. Real-time notifications (new/updated jobs) → URL_UPDATED
2. Archive notifications (removed jobs) → URL_DELETED
3. Bulk fix script (backlog of 433K URLs) → Both
Without coordination, the real-time service would eat the entire quota before the bulk fix could run a single request.
The Solution: Priority Queuing + Pause Mechanism
CREATE TABLE google_indexing_queue (
id SERIAL PRIMARY KEY,
job_id INTEGER,
url TEXT NOT NULL,
action VARCHAR(20) NOT NULL, -- 'URL_UPDATED' or 'URL_DELETED'
status VARCHAR(20) DEFAULT 'pending',
priority INTEGER DEFAULT 2, -- 1=DELETE (higher), 2=UPDATE
created_at TIMESTAMP DEFAULT NOW(),
sent_at TIMESTAMP,
UNIQUE(job_id, action)
);
Priority 1: Deletions. Google showing a dead job page is worse than not showing a new one. Deletions go first.
Priority 2: Updates. New pages can wait. Google will discover them via sitemap eventually.
The Pause Switch
async notifyUrlUpdated(jobId: number): Promise<IndexingResult> {
if (process.env.GOOGLE_INDEXING_PAUSE_REALTIME === 'true') {
return { success: true, error: 'Indexing paused' };
}
// ... send to Google
}
When we need the full 200/day quota for the bulk fix (clearing the backlog), we set GOOGLE_INDEXING_PAUSE_REALTIME=true. Real-time notifications are silently skipped, and the bulk fix gets 100% of the quota.
Timing: Quota Resets at Midnight Pacific
Google’s quota resets at midnight Pacific Time (8:00 AM UTC). Our bulk fix runs at 8:05 AM UTC — right after the reset:
// GoogleIndexingScheduler.ts
cron.schedule('5 8 * * *', () => {
// Run bulk fix with fresh 200 quota
exec('npx ts-node scripts/google-indexing-bulk-fix.ts');
});
The Math
| Queue | Pending |
|---|---|
| URL_DELETED | 371,014 |
| URL_UPDATED | 62,844 |
| Total | 433,858 |
At 200/day: 2,170 days (6 years). At 30,000/day (requested increase): 15 days.
The quota increase request is pending. In the meantime, the bulk fix processes 200 URLs daily, prioritizing deletions.
What Google Doesn’t Tell You
The quota is shared across all notification types. URL_UPDATED and URL_DELETED both count against the same 200/day limit.
Quota exceeded errors are permanent for the day. Once you hit 200, every subsequent request returns 429 until midnight Pacific.
The Indexing API doesn’t guarantee indexing. It’s a “please look at this” signal, not a command. Google can still decide not to index a page.
This system runs at MisuJob, managing 1M+ job listings across 175+ category pages.
Dealing with Google’s API quotas? What creative workarounds have you found?
