Smart Queue Management

Prioritize the URLs that matter without overwhelming origin.

Smart queues decide what gets warmed next. Cache Rocket balances discovery, priority paths, rate limits, and schedule windows so important URLs stay hot without turning your warmer into an accidental load test.

Highlights

• Prioritize entry URLs and critical paths
• Throttle with max URLs per minute
• Discover links within configured depth
• Avoid noisy or excluded URL patterns

Controlled discovery

Start from entry URLs or sitemaps, then expand by depth. Exclusions and regex options keep admin paths, infinite filters, and junk query strings out of the queue.

Throughput that respects origin

Max URLs per minute and request timeouts let you match hosting capacity. Raise throughput for large catalogs when you have headroom; keep it low for shared hosting.

How it works

• Seed the queue: Provide entry URLs, sitemaps, or both.
• Set guardrails: Configure depth, exclusions, and rate limits.
• Process continuously: Cache Rocket works through the queue on your interval.

Use cases

• Large catalogs: Warm high-value categories first while discovery continues.
• Shared hosting: Keep crawls gentle with strict per-minute caps.
• Campaign launches: Priority-warm landing pages before ads go live.

FAQ

Can I exclude URL patterns?

Yes. Use excluded URLs and, on supported plans, regex controls to keep the queue clean.

How deep should I crawl?

Start shallow for critical paths, then increase depth once health looks good and origin stays stable.