Concurrency & best practices

Throughput sizing, retry strategies, idempotency, webhook reliability, and tips for keeping cost predictable.

Chuger doesn't impose a hard cap on concurrent connections — what you can run in parallel is governed by your per-minute rate limit and your credit balance. This page collects best practices for using that headroom safely.

Throughput limits in practice

Your effective concurrency is bounded by the per-minute rate limit for the endpoint you're calling. For example, on Basic:

/v1/content allows 15 requests/minute → ~1 request every 4 seconds, or short bursts of 15 with appropriate spacing
/v1/prompt allows 8 requests/minute → bursts of 8, then wait

If you fire a burst that exceeds your per-minute allowance the excess returns 429. The window is sliding minute-by-minute, so spreading load is usually better than serial bursts. See Rate limits & quotas for the full per-plan matrix.

Use bulk endpoints for high-volume work

If you have many URLs to extract content from, prefer /v1/content/bulk over many parallel /v1/content calls.

Bulk:

Accepts up to 100 URLs (or a sitemap) in one request
Runs asynchronously by default — you get a 202 immediately
Delivers the results to an email or webhook when finished
Handles pacing internally so it doesn't trip your own rate limits
Only charges for successful extractions

Retry strategy

Build retries around three signals:

Signal	What to do
`429` status	Wait the `Retry-After` seconds, then retry the same request
`503` status	Exponential backoff (e.g. 1s, 2s, 4s, 8s) up to ~5 attempts. Different fallbacks may succeed
`5xx` other	Same exponential backoff

Don't retry 401, 402, 403, 404, or 422 — those are permanent until you fix the request or top up.

A minimal example:

async function fetchWithRetry(url, init, attempt = 0) {
  const res = await fetch(url, init);

  if (res.status === 429) {
    const retryAfter = Number(res.headers.get('Retry-After') ?? 5);
    await sleep(retryAfter * 1000);
    return fetchWithRetry(url, init, attempt + 1);
  }

  if (res.status >= 500 && attempt < 5) {
    await sleep(Math.min(30_000, 1000 * 2 ** attempt));
    return fetchWithRetry(url, init, attempt + 1);
  }

  return res;
}

Watch the warning headers

Two advisory headers let you back off proactively:

X-RateLimit-Warning: Approaching rate limit — slow down before you hit 429
X-Monthly-Quota-Warning: Approaching monthly quota — consider topping up before the next renewal

Treat these as soft pressure: pause spawning new workers, alert your team, or shed lower-priority traffic.

Idempotency

Requests are idempotent by design — the same URL or query will return the same shape of response. You can safely retry a 5xx or 429 failure without worrying about double-charging, because credits are only deducted on success.

For POST /v1/content/bulk, the resulting job_id lets you trace a single batch. Save it for support escalations.

URL guidance

To keep requests fast and consistent:

Use HTTPS when the target supports it
Stick to public URLs — Chuger rejects raw IPs and non-default ports
Keep URLs under 180 characters (most endpoints) or 500 for sitemap / callback URLs
URL-encode any special characters in query strings

Webhook reliability (bulk only)

If you set callback_url on /v1/content/bulk:

Three retries on non-2xx

Chuger will retry the webhook up to 3 times if your endpoint returns a non-2xx response.

30-second timeout per attempt

Each attempt times out at 30 seconds. Long-running handlers will be retried.

Route by event type

Inspect the X-Chuger-Webhook header to distinguish success from failure deliveries.

Acknowledge quickly

Respond as soon as you can — queue the payload, then process it asynchronously on your side, instead of waiting until heavy work completes.

Designing for cost

Always preview large jobs before launching them — see /v1/credits/preview-cost
Cache results on your side when you'll re-query the same URL or query repeatedly
Use /v1/serp/content with max_content set to the lowest value you can — every additional fetch costs credits
For /v1/serp/content, narrow with whitelist_domains or blacklist_domains so you don't pay to extract pages you'd filter out anyway

Was this page helpful?