Documentation
Introduction
Crawio returns reliable web data through one API. You send a URL; Crawio fetches it and returns a clean {status, headers, body}. One endpoint, one key, nothing to operate.
Base URL:
https://api.crawio.com
Quickstart
Grab your API key from the dashboard (API Key tab), then make your first call:
curl -X POST https://api.crawio.com/scrape \ -H "x-api-key: YOUR_API_KEY" \ -H "content-type: application/json" \ -d '{"url": "https://example.com"}'
Authentication
Every request is authenticated with your secret key in the x-api-key header. Keep it server-side; rotate it from the dashboard if exposed.
x-api-key: YOUR_API_KEY/scrapeFetch a URL and return the site's response. The only endpoint you need.
Body parameters
| Field | Type | Description |
|---|---|---|
| url * | string | The page to fetch. |
| method | string | HTTP method, default GET. POST/PUT supported. |
| headers | object | Request headers to forward. |
| body | string | Request body for POST/PUT. |
| timeout | number | Seconds to wait. |
| session_id | string | Sticky session. Pins this chain to one IP (see Sessions). |
| session_strict | boolean | With session_id: return 410 session_expired instead of moving the session to a new IP (see Sessions). |
| return_cookies | boolean | Return the site's session cookies so you can reuse them. |
Request
curl -X POST https://api.crawio.com/scrape \ -H "x-api-key: YOUR_API_KEY" -H "content-type: application/json" \ -d '{"url":"https://example.com/search","method":"GET"}'
Response
{ "status": 200, "headers": { "content-type": "text/html; charset=utf-8" }, "body": "<!doctype html>…", // the page (or JSON), parse this "finalUrl": "https://example.com/" }
A non-2xx from the target site comes back in status with the body intact; that’s the site’s response, not a Crawio error. See Status codes for Crawio’s own codes.
Files. When the response is a file (PDF, image, archive), body holds its bytes base64 encoded and the response adds "body_encoding": "base64". Text responses never carry that field. The body is always delivered uncompressed, so content-encoding and content-length are not in headers. Responses are limited to 20 MB.
import base64 data = r.json() if data.get("body_encoding") == "base64": open("file.pdf", "wb").write(base64.b64decode(data["body"])) else: html = data["body"]
Sessions & login
A session_id is a sticky session: every request carrying the same id is pinned to the same IP, so anything the site ties to that IP keeps working. Use one id per account, and keep each id on one worker thread.
To scrape behind a login, log in through Crawio with return_cookies, keep the cookies you get back, and send them on later requests. Crawio never stores your password. You hold the session.
# 1) log in through Crawio, asking for the cookies back r = requests.post("https://api.crawio.com/scrape", headers={"x-api-key": "YOUR_API_KEY", "x-session-id": "acct-1"}, json={"url": "https://site.com/login", "method": "POST", "headers": {"content-type": "application/x-www-form-urlencoded"}, "body": "user=...&pass=...", "return_cookies": True}).json() jar = r["session_cookies"] # [{"name": "sessionid", "value": "..."}, ...] cookie = "; ".join(f"{c['name']}={c['value']}" for c in jar) # 2) later requests: same session_id (same IP) + your cookies requests.post("https://api.crawio.com/scrape", headers={"x-api-key": "YOUR_API_KEY", "x-session-id": "acct-1"}, json={"url": "https://site.com/account", "headers": {"cookie": cookie}})
A session keeps its IP while you use it. If that IP gets blocked or the session sits idle for 15 minutes, the next request moves to a new IP. If the site ties your login to the IP, send "session_strict": true (or the header x-session-strict: 1) with the session: you get 410 session_expired instead, so you can log in again.
Concurrency with logins is bounded by accounts, not IPs. One account is one sticky session; running the same account from many IPs at once is what gets accounts flagged. For 8 concurrent logged-in requests, use 8 accounts and 8 session ids.
Anti-bot clearance cookies stay with Crawio and are excluded from responses. session_cookies contains only returnable site cookies, including login/session cookies. In Playground, enable Return session cookies to inspect, copy, or download them from the Cookies tab.
Status codes
| Code | Meaning |
|---|---|
| 200 | Success. The site's response is in the body. |
| 401 | Invalid API key. |
| 403 | Account suspended or disabled. |
| 409 | This Idempotency-Key was already used and its response returned. |
| 410 | Strict session expired: its IP is gone. Log in again. Not billed. |
| 413 | The response is over the 20 MB limit. Not billed. |
| 429 | Rate limit or quota exceeded. |
| 502 | Couldn't complete the fetch. |
| 503 | Capacity busy. Retry shortly. |
| 504 | Upstream timed out. |
Retry 429/502/503/504 with backoff. 401/403/409/410/413 are terminal. Billing: only successful responses count.
Idempotency-Key (recommended)
Send an Idempotency-Key header with a value unique to each request you make. If the connection drops or you get a 503, retry with the same key and you attach to the request already in progress instead of starting a second one. You get the same response, you are charged once, and no extra work is done on your behalf.
Without a key, a retry is simply a new request: it runs again and is billed again.
curl -X POST https://api.crawio.com/scrape \ -H "x-api-key: YOUR_API_KEY" \ -H "content-type: application/json" \ -H "Idempotency-Key: 8f14e45f-ea6f-4b1c-9f7a-2c1d3e4b5a60" \ -d '{"url": "https://example.com"}'
Reuse a key only to retry the same request. Once its response has been returned to you, that key is finished and reusing it returns 409.
What counts as billable
You are billed when Crawio reaches the site and the site gives a definitive answer about the resource you asked for. Everything else is free.
| Code | Meaning |
|---|---|
| 2xx | The site returned your content. Billable. |
| 404 | The site confirmed the resource no longer exists. Billable: the request went through and the site gave a definitive answer. |
| 403 and other 4xx | The site declined to serve this request, most often because it needs an authenticated session. Never billed. |
| 5xx | The site's own servers failed. Never billed. |
| Blocked | Crawio could not return a response for the request. Never billed. |
A site 4xx or 5xx is not a block
This distinction is easy to misread, and it is the one that matters. A 403, 404 or 5xx means the request went through: the response you received is the site’s own, delivered unblocked. The site chose to answer that particular request that way. None of these are a block, and none of them mean unblocking failed.
The most common reason is content that requires an authenticated session. A page loads normally, but a specific resource on it returns 403 to a signed-out visitor. That is the site’s access rule rather than a bot defence, and no unblocking service can change it. If you need content of that kind, see Sessions & login: a session carries your authenticated cookies across requests, and those resources then return 200.
Other common reasons: a 404 because the URL is no longer available, or a 5xx from the site’s own servers. Of these, only the 404 is billable, for the reason given below.
Which number tells you Crawio is working
Delivered is the share of your requests that came back with an unblocked response, whatever that response said. At or near 100%, unblocking is working as intended.
The distance between Delivered and your billable count is made up of requests the site declined or failed to answer, which are free. Treat it as a signal about the URLs you are sending: some may need a session.
Blocked is the number that reflects Crawio. If it starts to climb, raise it with us.
Why a 404 is billable and a 403 is not
Both cost us the same to produce, so the line is drawn on what the answer is worth to you.
A 404 is information. The site has confirmed the resource is gone, which is a result you can act on: you can drop that URL and stop asking for it. Nothing you or we could do differently would turn it into content.
A 403 is not an answer about the resource, it is the site declining to serve this particular request. The content usually still exists, and the same URL commonly returns 200 once the request carries an authenticated session. Charging for it would mean charging you twice for one piece of data, since you would fetch it again with a session. See Sessions & login.
A 5xx is the site failing rather than answering, so it is free for the same reason a block is.
Scrapy
Drop-in downloader middleware: matched domains route through Crawio; spiders stay normal.
DOWNLOADER_MIDDLEWARES = {"scrapy_replay_proxy.ReplayProxyMiddleware": 1000} REPLAY_PROXY_URL = "https://api.crawio.com" REPLAY_PROXY_API_KEY = "YOUR_API_KEY" REPLAY_PROXY_DOMAINS = ["example.com"] # hosts to route through Crawio REPLAY_PROXY_SESSION = "job-1" # optional sticky session RETRY_ENABLED = True RETRY_HTTP_CODES = [429, 502, 503, 504]