Crawio API

Introduction

Crawio returns reliable web data through one API. You send a URL; Crawio fetches it and returns a clean {status, headers, body}. One endpoint, one key, nothing to operate.

Base URL:

https://api.crawio.com

Quickstart

Grab your API key from the dashboard (API Key tab), then make your first call:

curl -X POST https://api.crawio.com/scrape \
  -H "x-api-key: YOUR_API_KEY" \
  -H "content-type: application/json" \
  -d '{"url": "https://example.com"}'

Integrate with AI

Paste this prompt into your coding agent. It works with any language or framework: the agent reads your project first, asks before it changes existing code, keeps your key in the CRAWIO_API_KEY environment variable, adds tests and makes one real request to check.

AI promptAny language, any framework

Integrate the Crawio web scraping API into this project, so every web page this project fetches goes through Crawio. Crawio fetches any page for me: proxies and anti-bot handling happen on its side. I send a URL and get back the site's status, headers and body.
1. Look at the project first
Before changing anything, read the project and tell me:
•the language, framework and HTTP client it already uses (follow those conventions for config, errors, logging and tests; do not add a new HTTP library if one is already there);
•every place that fetches web pages today (requests, httpx, fetch, axios, curl, Guzzle, net/http, Scrapy, Puppeteer, Playwright, Selenium or anything else);
•what you plan to add, and which of those places you plan to switch to Crawio.
Then wait for my go before switching existing fetching code. Adding the new client module on its own is fine.
If the project uses Scrapy, do not write a client: install the scrapy-crawio package and enable its middleware in settings.py: DOWNLOADER_MIDDLEWARES = {"scrapy_crawio.CrawioMiddleware": 1000}, CRAWIO_API_KEY set from the environment variable, CRAWIO_DOMAINS listing the hosts to route, RETRY_HTTP_CODES = [429, 502, 503, 504].
2. The API key
•Read the key from the environment variable CRAWIO_API_KEY. Never hard-code it, print it, log it or commit it.
•Add CRAWIO_API_KEY= to .env.example (or the project's equivalent) and tell me where to put the real key. I get it from the API Key tab of the Crawio dashboard.
3. How the API works
One endpoint: POST https://api.crawio.com/scrape
Headers: x-api-key: <key>, content-type: application/json, Idempotency-Key: <a new random id per request>
JSON body:
•url (string, required): the full http:// or https:// address of the page.
•method (string): GET by default; POST, PUT, PATCH, DELETE or HEAD.
•headers (object): headers to send to the site, for example a cookie or content-type. Do not add User-Agent or other browser headers to get past bot checks; Crawio handles that.
•body (string): the request body for POST or PUT, sent as-is.
•timeout (number): seconds Crawio waits for the page. Default 180.
•country (string): a two-letter code (DE, FR...) to fetch the page from that country. Leave it out unless the page really differs by country: it costs more credits.
•session_id (string): a sticky session; requests with the same id keep the same IP. Only for logins or multi-step flows.
•session_strict (boolean): with a session, answer 410 instead of moving to a new IP.
•return_cookies (boolean): return the site's cookies in session_cookies, for logins.
•render (boolean, beta, early access): load the page in a real browser and return the HTML after its JavaScript ran. Rendered pages cost more credits than standard requests; pricing for this beta is not final, and the credits used are in cost on each answer. Only for pages that are empty or partial without it. Early access: available on request; until it is on for the account, render and actions are ignored and the answer has no rendered field. Render options: wait (number, ms after load, 0 to 30000), wait_for (string, a CSS selector to wait for, up to 20 s; :has-text("...") matches by text), wait_until (networkidle by default, load or domcontentloaded), block_resources (boolean, default true). GET only; a render runs up to 120 seconds (or until timeout when lower): send timeout 120 or more; a render that runs out of time is a 504, not billed.
•actions (array, beta, early access, with render): steps run in order before the page is read, each an object with one of: click (selector), wait (ms, up to 10000), wait_for (selector), scroll ("bottom" or pixels), fill (selector, with value), press (Enter, Tab, Escape, ArrowDown...), evaluate (JavaScript). "optional": true lets the next steps run when that step fails. Up to 20 steps and 60 seconds. A failed step stops the steps and the answer carries actions_error ("step 2: no element matches .more."), billed as a rendered page.
Example request body: {"url": "https://books.toscrape.com/", "method": "GET"}
A 200 from Crawio returns JSON:
•status: the SITE's HTTP status. A site 404 or 500 still arrives as a 200 from Crawio; check this field, not only the HTTP status of the call.
•headers: the site's response headers.
•body: the page as text, or a file's bytes base64 encoded when body_encoding is "base64" (decode it then).
•finalUrl: where the request ended after redirects (only when it moved).
•request_id: the reference for support. cost: the credits it used.
•rendered: true when the page was rendered (render). actions_error: only when a render's step failed or its steps did not run.
The headers X-Crawio-Request-Id and X-Crawio-Cost carry the same two values on every answer.
Crawio's own errors: the HTTP status says which, and the JSON body has error (a stable code), message, and request_id, retry_after and retry (true or false) when they apply.
•400 country_unavailable: that country cannot be served. 400 render_unavailable: that render cannot be done (not billed).
•401: missing or invalid key.
•403: not allowed on this account; the message says why.
•409 idempotency_key_already_delivered: this key's response was already handed out, for example while the connection was dropping.
•410 session_expired: a strict session lost its IP; log in again.
•413: the response is over the size limit (50 MB on paid plans, 10 MB on the free plan, 20 MB on some sites served from residential IPs); the message says which. A response over 10 MB uses 5 credits (10 on sites served from residential IPs), and an account fetches up to 2 of them at a time.
•422 invalid_request: the request is malformed; the message says what to fix.
•429: a limit; error says which. concurrency_limit and rate_limited clear within seconds. daily_quota_exceeded, weekly_quota_exceeded, monthly_quota_exceeded, failure_budget_exhausted and credits_used_up last until the plan resets or is upgraded (their Retry-After, when sent, is hours or days). key_credit_limit is this API key's own credit limit, set on the API Key page: not retryable until the key's period resets (Retry-After) or the limit is raised there; other keys keep working.
•502: the page could not be fetched. 503: busy or restarting. 504: timed out. Their body says retry true or false.
Billing: only the site's real answers bill, a 2xx or its 404 or 410; Crawio's own errors never do.
4. Build one small client
Write one module the rest of the code calls, in the project's style. It must:
•send the request above and return status, headers, body, finalUrl, cost and request_id;
•set the HTTP client timeout above the request timeout (200 seconds for the default 180);
•retry, up to 4 times, only these: 429 with error concurrency_limit or rate_limited; 502, 503 or 504 whose body says retry true; 502, 503 or 504 with no JSON body (a gateway error); and dropped connections. Every other error is final: return it with its message, do not retry;
•wait Retry-After seconds when it is given, otherwise back off 1, 2, 4 then 8 seconds, each plus a random 0 to 1 second;
•reuse the SAME Idempotency-Key for the retries of one request, so a retry attaches to it and is never billed twice;
•if a retry after a dropped connection gets 409 idempotency_key_already_delivered, the first answer was lost in transit: send the request once more with a NEW Idempotency-Key (that one can bill again), and do not repeat this more than once;
•keep at most CRAWIO_CONCURRENCY requests running at once (read it from the environment, default 2, and tell me to set it to my plan's concurrent requests, shown on the Subscription tab);
•decode a base64 body when body_encoding is "base64";
•log request_id, status and cost for each call, never the key.
5. Switch the project over
After my go, change the places you listed to call this client. Keep their parsing and business logic as it is: only the fetching changes. List any proxy rotation, retry-on-block loops or headless browsers that exist only to get past blocks; Crawio replaces them, so remove them once I agree.
6. Test
•Add unit tests for the client with the HTTP call mocked: success, a site 404 inside a 200, a 429 concurrency_limit retried after Retry-After, a 429 monthly_quota_exceeded returned at once without retrying, a 503 with retry true retried with the same Idempotency-Key, a 502 with retry false not retried, a 409 after a dropped connection resent once with a new key, a base64 file.
•Then make ONE real request to https://books.toscrape.com/ and show me the status, cost and request_id. Do not run anything bigger without asking me: real requests use my credits.
Guardrails
•Do not run load tests, loops or crawls against the live API.
•Do not delete or rewrite code you were not asked to touch. If a change would, stop and ask me.
Full reference: https://crawio.com/docs

Scroll to read more

Authentication

Every request is authenticated with your secret key in the x-api-key header. Keep it server-side; rotate it from the dashboard if exposed.

x-api-key: YOUR_API_KEY

POST/scrape

Fetch a URL and return the site's response. The only endpoint you need.

Body parameters

Field	Type	Description
url *	string	The page to fetch.
method	string	GET (default), POST, PUT, PATCH, DELETE or HEAD.
headers	object	Request headers to forward.
body	string	Request body for POST/PUT.
timeout	number	Seconds to wait for the result. Default 180.
country	string	Two-letter country code (ISO 3166-1 alpha-2, e.g. DE, case-insensitive) the page is fetched from: 184 countries and 33 regions (see Countries). Without it the best exit for the site is used.
session_id	string	Sticky session: pins this chain of requests to one IP (see Sessions). The same as the x-session-id header; send either.
session_strict	boolean	With a session: answer 410 session_expired instead of moving the session to a new IP (see Sessions). The same as the header x-session-strict: 1.
return_cookies	boolean	Return the site's session cookies so you can reuse them.
render	boolean	Beta, early access: load the page in a real browser and return the HTML after its JavaScript ran. Its options wait, wait_for, wait_until and block_resources are in Render JavaScript.
actions	array	Beta, early access, with render: steps the browser runs before the page is read, like a click or a scroll (see Render JavaScript).

Request headers

Header	Value	Description
x-api-key *	string	Your API key (see Authentication).
x-session-id	string	A sticky session id: the same as the session_id field.
x-session-strict	1	The same as the session_strict field.
Idempotency-Key	string	A value unique to each request, so a retry attaches to the request instead of running it twice (see Status codes).

Request

curl -X POST https://api.crawio.com/scrape \
  -H "x-api-key: YOUR_API_KEY" -H "content-type: application/json" \
  -d '{"url":"https://example.com/search","method":"GET"}'

Response

{
  "status": 200,
  "ms": 929,
  "finalUrl": "https://example.com/",
  "body": "<!doctype html><html lang=\"en\"><head><title>Example Domain</title>…",
  "headers": {
    "content-type": "text/html",
    "date": "Fri, 18 Sep 2026 20:10:49 GMT",
    "last-modified": "Tue, 15 Sep 2026 23:38:37 GMT"
  },
  "request_id": "req_0GCW27J",
  "cost": 1
}

Field	Type	Description
status	number	The site's HTTP status.
headers	object	The site's response headers.
body	string	The page, or the JSON the site returned. A file's bytes base64 encoded when body_encoding is base64 (see Files below).
body_encoding	string	Only for a file (PDF, image, archive): base64.
finalUrl	string	Where the request ended when it was redirected or its address was normalized. Absent otherwise.
ms	number	How long the request took, from the moment Crawio received it to the answer, in milliseconds.
request_id	string	The reference to quote when you contact support.
cost	number	Credits the request used, or 0 when it was not billed. Most sites use 1.
session_cookies	array	Only with return_cookies: the site's session cookies (see Sessions).
captcha	object	Only when Crawio solved a form CAPTCHA for the request.

Every response names its request and its cost. request_id is the reference to quote when you contact support, and cost is the credits the request used, 1 on most sites, or 0 when it was not billed. The same two values arrive as the headers X-Crawio-Request-Id and X-Crawio-Cost on every answer, errors included, and an error body carries request_id too.

A non-2xx from the target site comes back in status with the body intact; that’s the site’s response, not a Crawio error. See Status codes for Crawio’s own codes.

Files. When the response is a file (PDF, image, archive), body holds its bytes base64 encoded and the response adds "body_encoding": "base64". Text responses never carry that field. The body is always delivered uncompressed, so content-encoding and content-length are not in headers. Responses are limited to 50 MB on paid plans and 10 MB on the free plan (20 MB on some sites served from residential IPs). A response over 10 MB uses 5 credits, or 10 on sites served from residential IPs. Your account fetches up to 2 responses over 10 MB at a time; the others wait their turn.

import base64
data = r.json()
if data.get("body_encoding") == "base64":
    open("file.pdf", "wb").write(base64.b64decode(data["body"]))
else:
    html = data["body"]

Render JavaScript

Beta, early access

Early access: available on request. Contact us to enable it for your account. Until it is on, render and actions are ignored: the page is fetched as a normal request and billed as one, and the answer has no rendered field. Check rendered to know which one you got.

Some pages build their content with JavaScript after they load, so a plain request returns an empty or partial page. With "render": true a real browser opens the page, runs its JavaScript and returns the HTML it shows. With actions it can also click, type and scroll first, for example to open a section or run a search.

Render options

Field	Type	Description
render *	boolean	true turns rendering on. Default false.
wait	number	Milliseconds to wait once the page counts as loaded, a whole number from 0 to 30000. Default 0.
wait_for	string	A CSS selector, up to 300 characters, to wait for, up to 20 seconds: for example .results. :has-text("...") matches by text, for example h1:has-text('Travel'). When nothing matches in time the page comes back as it is then.
wait_until	string	When the page counts as loaded: networkidle (default: the page has stopped loading data), load (the page and its files loaded) or domcontentloaded (the HTML is ready: faster, for pages whose content is in the HTML).
block_resources	boolean	true (default): images, fonts and media are not downloaded, which is faster. false loads everything.
actions	array	Steps run in the page before it is read, up to 20 (see Actions below).
timeout	number	Seconds to wait for the answer, as on every request (default 180). A render runs up to 120 seconds, or until your timeout when it is lower: send 120 or more for the full time. A render that runs out of time answers 504 timeout and is not billed.

Rendering works with GET requests, and with country, session_id and return_cookies. Your headers are sent with the page.

Example: a page built by JavaScript

curl -X POST https://api.crawio.com/scrape \
  -H "x-api-key: $CRAWIO_API_KEY" \
  -H "content-type: application/json" \
  -d '{"url": "https://quotes.toscrape.com/js/", "render": true, "wait_for": ".quote", "timeout": 120}'

Response

The usual answer (see POST /scrape), with these fields for a render:

Field	Type	Description
body	string	The HTML the page shows once loaded, after the last action. A file (PDF, image, JSON) comes back as the file itself.
status	number	The page's own HTTP status. When an action opened a new page, that page's status.
headers	object	The page's own response headers.
finalUrl	string	The page's address at the end, when it differs from url (a redirect, or an action that opened another page or changed the address).
rendered	boolean	true when the page was rendered. Absent when it was fetched as a normal request (rendering not on for your account).
actions_error	string	Only when a step failed or the steps did not run: which step and why, for example step 2: no element matches .load-more.
cost	number	Credits used by this request, actions included (see Credits and errors below).

Actions

With rendering on, actions is a list of steps the browser runs in order once the page has loaded: open a section, press a “load more” button, scroll, type into a search box. The answer is the page after the last step. Each step is a JSON object with exactly one action:

Step	Fields	What it does
click	{"click": "<selector>"}	Clicks the first element that matches. Its element is looked for up to 5 seconds (1 second when optional). When the click opens a new page, the next step starts once that page has loaded.
wait	{"wait": 1500}	Waits this many milliseconds, 0 to 10000.
wait_for	{"wait_for": "<selector>"}	Waits until an element that matches is in the page, up to 20 seconds. :has-text("...") matches by text.
scroll	{"scroll": "bottom"} or {"scroll": 800}	Scrolls to the bottom of the page (endless pages load their next items), or by this many pixels, -100000 to 100000 (negative scrolls up).
fill	{"fill": "<selector>", "value": "text"}	Types value, up to 2000 characters, into a text field. Its element is looked for up to 5 seconds (1 second when optional).
press	{"press": "Enter"}	Presses a key: Enter, Tab, Escape, Backspace, Delete, Space, ArrowDown, ArrowUp, ArrowLeft, ArrowRight, PageDown, PageUp, Home or End.
evaluate	{"evaluate": "<JavaScript>"}	Runs your JavaScript in the page, up to 2000 characters: an expression, statements, or a promise, which is awaited. What it returns is not used.
optional	"optional": true	On any step: when it fails, the next steps still run, for example a banner that is not always there. Default false.

Selectors are CSS selectors of up to 300 characters, for example #search, a.next or button[aria-label='Open']. :has-text("...") matches by text.
After a click, fill, press, scroll or evaluate, the next step starts once what it started has settled: a new page has loaded, or the data the page asked for has arrived. A step that starts nothing takes about half a second.
Limits: up to 20 steps; the wait steps add up to 60 seconds at most; all steps together get up to 60 seconds, within the render’s own time.
Every request runs in a fresh browser of its own: nothing a step does reaches another request or account.

Example: open a section, then wait for its content

curl -X POST https://api.crawio.com/scrape \
  -H "x-api-key: $CRAWIO_API_KEY" \
  -H "content-type: application/json" \
  -d '{"url": "https://books.toscrape.com/", "render": true, "timeout": 120,
       "actions": [{"click": "a[href*=travel_2]"},
                   {"wait_for": "h1:has-text(\"Travel\")"}]}'

Example: fill a search box and read the results

curl -X POST https://api.crawio.com/scrape \
  -H "x-api-key: $CRAWIO_API_KEY" \
  -H "content-type: application/json" \
  -d '{"url": "https://www.scrapethissite.com/pages/forms/", "render": true, "timeout": 120,
       "actions": [{"click": "#accept-cookies", "optional": true},
                   {"fill": "input#q", "value": "Boston"},
                   {"press": "Enter"},
                   {"wait_for": "tr.team"}]}'

When a step fails

The steps stop at the first one that fails (unless it is optional), and the answer is the page as it was then, with actions_error naming the step and why. It is billed as a rendered page: the page was loaded and returned.

import os, requests

data = requests.post("https://api.crawio.com/scrape",
                     headers={"x-api-key": os.environ["CRAWIO_API_KEY"]},
                     json={"url": "https://books.toscrape.com/", "render": True, "timeout": 120,
                           "actions": [{"click": "a[href*='travel_2']"}, {"click": ".load-more"}]},
                     timeout=200).json()
if data.get("actions_error"):
    print("steps stopped:", data["actions_error"])   # step 2: no element matches .load-more.
html = data["body"]                                  # the page after step 1

actions_error		Meaning
step N: no element matches X.		Nothing matched the selector in time. Put a wait_for before the step when its element comes later.
step N: no element matching X appeared within 20 s.		A wait_for that never matched.
step N: X is not a valid CSS selector.		Fix the selector (an id that starts with a digit needs [id="2015"]).
step N: X is not a text field that can be filled.		fill works on text inputs and text areas.
step N: X could not be clicked.		The element was there but did not take the click.
step N: evaluate failed: ...		Your JavaScript threw; the page's own error message follows.
step N: evaluate did not finish in time.		Your JavaScript (or its promise) ran past the time left.
step N did not run: the actions ran out of time.		All steps share 60 seconds within the render's own time.
The actions did not run: the page answered HTTP 404.		The page itself was an error; the steps run only on a page.
The actions did not run: the answer is a file, not a page.		The url returned a file (PDF, image, JSON).

Credits and errors

Price. Rendered pages cost more credits than standard requests. Pricing for this beta is not final; the credits used are shown in each response and in your Request log. Actions cost nothing on top of the render, and a page whose steps stopped at an error is billed as a rendered page. A request that fails (the page could not be loaded, a timeout, a Crawio error) is not billed; a site’s own 404 or 410 bills like any page.

At once. Your account renders 2 pages at a time on the Free and Starter plans and more on higher plans; further renders wait their turn.

Errors. A render option or step that is not right is a 422 invalid_request whose message says what to fix, and nothing is billed, for example step 1: unknown step type "hover". Use click, wait, wait_for, scroll, fill, press or evaluate. or render works with GET requests only. Actions without render: actions need "render": true. When a request cannot be rendered (for example a country we cannot render from), the answer is 400 render_unavailable, not billed.

GET/account

Your plan, the credits left in this billing period and your concurrent requests, so your code can check before a big job. It is free: it is not a request and it does not show in your request log.

curl https://api.crawio.com/account \
  -H "x-api-key: YOUR_API_KEY"

Field	Type	Description
plan	string	Your plan.
status	string	active, or suspended when requests are paused on your account.
credits.allowance	number	Credits this billing period holds. null when your plan has no monthly limit.
credits.used	number	Credits used in this billing period. On the free plan, since you signed up.
credits.remaining	number	Credits left. A request in progress counts once it finishes.
credits.one_time	boolean	true on the free plan: its credits are given once and do not refill.
period.start, period.renews	string	When this billing period started and when your plan renews (UTC). period is null on the free plan.
concurrency.limit	number	How many requests can run at the same time.
concurrency.in_use	number	How many are running now.

A key that is missing or not valid gets the same 401 as a scrape.

Countries

Add country to a request and the page is fetched from that country, the way a visitor there sees it. One two-letter code (ISO 3166-1 alpha-2, case-insensitive), 184 countries and 33 regions. Use it for:

local prices, currencies and stock
search results, listings and ads as one market sees them
pages that only open inside a country

Leave it out when the page is the same everywhere: Crawio then picks the best exit for the site.

curl -X POST https://api.crawio.com/scrape \
  -H "x-api-key: YOUR_API_KEY" -H "content-type: application/json" \
  -d '{"url":"https://example.com/pricing","country":"DE"}'

Popular countries

United StatesUS

United KingdomGB

GermanyDE

FranceFR

SpainES

ItalyIT

NetherlandsNL

CanadaCA

AustraliaAU

BrazilBR

MexicoMX

IndiaIN

JapanJP

All 184 countries and 33 regions

AndorraAD

United Arab EmiratesAE

AfghanistanAF

Antigua & BarbudaAG

AnguillaAI

AlbaniaAL

ArmeniaAM

AngolaAO

ArgentinaAR

AustriaAT

AustraliaAU

ArubaAW

AzerbaijanAZ

Bosnia & HerzegovinaBA

BarbadosBB

BangladeshBD

BelgiumBE

Burkina FasoBF

BulgariaBG

BahrainBH

BurundiBI

BeninBJ

BermudaBM

BruneiBN

BoliviaBO

Caribbean NetherlandsBQ

BrazilBR

BahamasBS

BhutanBT

BotswanaBW

BelarusBY

BelizeBZ

CanadaCA

Congo - KinshasaCD

Central African RepublicCF

Congo - BrazzavilleCG

SwitzerlandCH

Côte d’IvoireCI

ChileCL

CameroonCM

ChinaCN

ColombiaCO

Costa RicaCR

Cape VerdeCV

CuraçaoCW

CyprusCY

CzechiaCZ

GermanyDE

DjiboutiDJ

DenmarkDK

DominicaDM

Dominican RepublicDO

AlgeriaDZ

EcuadorEC

EstoniaEE

EgyptEG

EritreaER

SpainES

EthiopiaET

FinlandFI

FijiFJ

Falkland IslandsFK

FranceFR

GabonGA

United KingdomGB

GrenadaGD

GeorgiaGE

French GuianaGF

GuernseyGG

GhanaGH

GibraltarGI

GreenlandGL

GambiaGM

GuineaGN

GuadeloupeGP

Equatorial GuineaGQ

GreeceGR

GuatemalaGT

GuamGU

Guinea-BissauGW

GuyanaGY

Hong Kong SAR ChinaHK

HondurasHN

CroatiaHR

HaitiHT

HungaryHU

IndonesiaID

IrelandIE

IsraelIL

Isle of ManIM

IndiaIN

British Indian Ocean TerritoryIO

IraqIQ

IcelandIS

ItalyIT

JerseyJE

JamaicaJM

JordanJO

JapanJP

KenyaKE

KyrgyzstanKG

CambodiaKH

ComorosKM

St. Kitts & NevisKN

South KoreaKR

KuwaitKW

Cayman IslandsKY

KazakhstanKZ

LaosLA

LebanonLB

St. LuciaLC

LiechtensteinLI

Sri LankaLK

LiberiaLR

LesothoLS

LithuaniaLT

LuxembourgLU

LatviaLV

LibyaLY

MoroccoMA

MonacoMC

MoldovaMD

MontenegroME

St. MartinMF

MadagascarMG

North MacedoniaMK

MaliML

Myanmar (Burma)MM

MongoliaMN

Macao SAR ChinaMO

MartiniqueMQ

MauritaniaMR

MontserratMS

MaltaMT

MaldivesMV

MalawiMW

MexicoMX

MalaysiaMY

NamibiaNA

New CaledoniaNC

NigerNE

NigeriaNG

NicaraguaNI

NetherlandsNL

NorwayNO

NepalNP

NauruNR

New ZealandNZ

OmanOM

PanamaPA

PeruPE

French PolynesiaPF

Papua New GuineaPG

PhilippinesPH

PakistanPK

PolandPL

Puerto RicoPR

Palestinian TerritoriesPS

PortugalPT

PalauPW

ParaguayPY

QatarQA

RéunionRE

RomaniaRO

SerbiaRS

RussiaRU

RwandaRW

Saudi ArabiaSA

SeychellesSC

SudanSD

SwedenSE

SingaporeSG

St. HelenaSH

SloveniaSI

SlovakiaSK

Sierra LeoneSL

San MarinoSM

SenegalSN

SomaliaSO

SurinameSR

South SudanSS

São Tomé & PríncipeST

El SalvadorSV

Sint MaartenSX

SyriaSY

EswatiniSZ

Turks & Caicos IslandsTC

ChadTD

TogoTG

ThailandTH

TajikistanTJ

Timor-LesteTL

TurkmenistanTM

TunisiaTN

TongaTO

TürkiyeTR

Trinidad & TobagoTT

TaiwanTW

TanzaniaTZ

UkraineUA

UgandaUG

United StatesUS

UruguayUY

UzbekistanUZ

St. Vincent & GrenadinesVC

VenezuelaVE

British Virgin IslandsVG

U.S. Virgin IslandsVI

VietnamVN

VanuatuVU

Wallis & FutunaWF

SamoaWS

YemenYE

MayotteYT

South AfricaZA

ZambiaZM

ZimbabweZW

The Playground has the same list in its country picker, so you can check a page from any country before you code it.

Answers

Code	Meaning
400	error country_unavailable: the country cannot be served. Answered at once, before anything runs, with available: every code you can use. Not billed.
403	error country_upgrade_required: the free trial's 5 geotargeted requests are used. Any paid plan targets every country. Not billed.
422	error invalid_request: the code is not two letters.

Good to know

Billing. A request that targets a country other than the US uses 10 credits, with the first 500 KB of the answer included and 5 credits per further 100 KB. With US or no country it costs what any request does. Only successful requests bill, and cost shows the credits each one used.
Sessions. A sticky session keeps its country: send the same country on every request of the session.
A site that refuses a country. Some sites turn away visitors from some countries. A request that fails is not billed: try another country, or leave country out.

Sessions & login

A session_id is a sticky session: every request carrying the same id is pinned to the same IP, so anything the site ties to that IP keeps working. Use one id per account, and keep each id on one worker thread.

To scrape behind a login, log in through Crawio with return_cookies, keep the cookies you get back, and send them on later requests. Crawio never stores your password. You hold the session.

# 1) log in through Crawio, asking for the cookies back
r = requests.post("https://api.crawio.com/scrape",
  headers={"x-api-key": "YOUR_API_KEY", "x-session-id": "acct-1"},
  json={"url": "https://site.com/login", "method": "POST",
        "headers": {"content-type": "application/x-www-form-urlencoded"},
        "body": "user=...&pass=...",
        "return_cookies": True}).json()

jar = r["session_cookies"]                     # [{"name": "sessionid", "value": "..."}, ...]
cookie = "; ".join(f"{c['name']}={c['value']}" for c in jar)

# 2) later requests: same session_id (same IP) + your cookies
requests.post("https://api.crawio.com/scrape",
  headers={"x-api-key": "YOUR_API_KEY", "x-session-id": "acct-1"},
  json={"url": "https://site.com/account", "headers": {"cookie": cookie}})

A session keeps its IP while you use it. If that IP gets blocked or the session sits idle for 15 minutes, the next request moves to a new IP. If the site ties your login to the IP, send "session_strict": true (or the header x-session-strict: 1) with the session: you get 410 session_expired instead, so you can log in again.

Concurrency with logins is bounded by accounts, not IPs. One account is one sticky session; running the same account from many IPs at once is what gets accounts flagged. For 8 concurrent logged-in requests, use 8 accounts and 8 session ids.

Anti-bot clearance cookies stay with Crawio and are excluded from responses. session_cookies contains only returnable site cookies, including login/session cookies. In Playground, enable Return session cookies to inspect, copy, or download them from the Cookies tab.

Status codes

Code	Meaning
200	Success. The site's response is in the body.
400	The request asked for a country that cannot be served (error country_unavailable, with available: every code you can use; see Countries), or a render that cannot be done (error render_unavailable; see Render JavaScript). Not billed.
401	No API key (error missing_api_key) or a key that is not valid (error invalid_api_key).
403	Account suspended or disabled, this account has no free trial (error trial_unavailable; upgrade to keep going), the site is not available on your plan (error domain_not_allowed; message says why), or the free trial's geotargeted requests are used (error country_upgrade_required). Not billed.
409	This Idempotency-Key was already used and its response returned.
410	Strict session expired: its IP is gone. Log in again. Not billed.
413	The response is over the size limit (50 MB on paid plans, 10 MB on the free plan, 20 MB on some sites served from residential IPs; the message says which), or the request body you send is over 2 MB. Not billed.
422	The request is not valid: not JSON, a field is missing or wrong, the url is not a full http:// or https:// address, or the method is not one of the six above (error invalid_request; the message says what to fix, for example "url is required."). Checked before any plan limit. Not billed.
429	A plan limit was reached. The body says which one. concurrency_limit and rate_limited clear within seconds: wait Retry-After and retry. daily_quota_exceeded, weekly_quota_exceeded, monthly_quota_exceeded, failure_budget_exhausted and credits_used_up last until the plan resets or is upgraded, so do not retry them; their Retry-After, when sent, is hours or days. key_credit_limit is this API key's own credit limit, set on the API Key page: it lasts until the key's period resets (Retry-After says when) or the limit is raised, so do not retry it; other keys keep working. The body carries the credits used and the limit. Not billed. See Key limits and alerts.
502	Couldn't complete the fetch. error site_unreachable means the site does not exist (its domain has no DNS record): check the URL and do not retry. That one is not a Crawio error.
503	Capacity busy. Retry shortly.
504	Upstream timed out.

Retry 429/502/503/504 with backoff. 400/401/403/409/410/413/422 are terminal. Billing: only delivered pages (2xx) and the site's own 404 or 410 count; these error codes are never billed.

Every error has the same shape: a JSON body with error, a stable code to branch on, and message, a sentence you can show or log. request_id and retry_after are there when they apply. detail repeats the older form of the error and stays for existing integrations.

Concurrent requests

Your plan sets how many requests can run at the same time; the Subscription tab shows the number. A request sent while that many are still running is refused at once with 429, error concurrency_limit, and a Retry-After header in seconds. It is not queued and not billed. Keep at most that many requests open, or retry after the wait.

Idempotency-Key (recommended)

Send an Idempotency-Key header with a value unique to each request you make. If the connection drops or you get a 503, retry with the same key and you attach to the request already in progress instead of starting a second one. You get the same response, you are charged once, and no extra work is done on your behalf.

Without a key, a retry is simply a new request: it runs again and is billed again.

curl -X POST https://api.crawio.com/scrape \
  -H "x-api-key: YOUR_API_KEY" \
  -H "content-type: application/json" \
  -H "Idempotency-Key: 8f14e45f-ea6f-4b1c-9f7a-2c1d3e4b5a60" \
  -d '{"url": "https://example.com"}'

Reuse a key only to retry the same request. Once its response has been returned to you, that key is finished and reusing it returns 409. A 502, 503 or 504 says in its body whether a retry can help (retry true or false). If the connection dropped while the answer was on its way, a retry can get that 409: send the request once more with a new key (it is billed again).

GET/plugin/google/maps/search

Google Maps

Search Google Maps and get the places back as clean JSON: name, rating, reviews, category, address, phone, website, opening state, coordinates and attributes. 5 credits per page of 20 places, only when places come back. Try it in the Playground under Plugins.

Field	Type	Description
q *	string	What to search for, like coffee shops austin.
ll	string	Optional: an exact map spot to search around, as coordinates in any common form (30.2672, -97.7431 or 30°16'01.9"N 97°44'35.2"W, with an optional zoom like 15z) or a Google Maps link. Leave it out to place the search by the words in q, like coffee shops in Austin, TX; a place name sent here is added to q. search_parameters.location says how it was read.
start	number	Result offset for the next pages: 0, 20, 40 and so on. 20 places per page.
pages	number	How many pages from start in one search, 1 (default) to 10. Each page with places is billed; the run stops at the first empty page, which is free. search_information.next_start says where the next page starts, null when there are no more.
session	string	Keeps every page on one IP. Pages of one call always share one; to fetch pages in separate calls, send the session a first call returned (search_parameters.session) so they stay consistent.
hl	string	Language of the results, like en (default), de or pt-BR.
gl	string	Google country, a two-letter code like us (default) or de.

curl "https://api.crawio.com/plugin/google/maps/search?q=coffee+shops+austin" \
  -H "x-api-key: YOUR_API_KEY"

{
  "search_parameters": {"engine": "google_maps", "q": "coffee shops austin", "start": 0, "hl": "en", "gl": "us"},
  "search_information": {"query_displayed": "coffee shops austin", "results": 20},
  "local_results": [
    {
      "position": 1,
      "title": "Cherrywood Coffeehouse",
      "place_id": "ChIJscwUlfO1RIYRmKPeAAr1Zts",
      "gps_coordinates": {"latitude": 30.2935632, "longitude": -97.7160212},
      "rating": 4.5,
      "reviews": 1269,
      "type": "Coffee shop",
      "address": "1400 E 38th 1/2 St, Austin, TX 78722",
      "phone": "(512) 538-1991",
      "website": "https://cherrywoodcoffeehouse.com/",
      "open_state": "Open · Closes 10 PM",
      "extensions": {"accessibility": ["Wheelchair accessible entrance"]}
    }
  ],
  "request_id": "req_0GCW27J",
  "cost": 5
}

Fields a place does not have are left out. A page that does not load is asked again on a new IP, free, and the search goes on there: search_information.retried lists those pages and search_parameters.session is the session to keep using. If a page still does not load, or your credits run out during a search, the answer has the pages before it and search_information.stopped. To get the rest, call again with start set to stopped.start, the same session and a new Idempotency-Key. A search counts as one of your concurrent requests until it answers. When Google sends a page without review counts, it is asked again; a page that still has none is not billed and is listed in search_information.reviews_missing. Errors have the same shape as the scrape API; a search Google answers without places data is 502 plugin_no_results and is not billed.

GET/plugin/google/maps/reviews

Google Maps Reviews

Get a place's reviews as clean JSON: rating, date, text, likes, photos, the owner's response, the reviewer and, for restaurants, the visit details. The first page also has the place (name, address, rating, review count) and its review topics with how often each is mentioned. 5 credits per page of up to 20 reviews, only when reviews come back. Find a place's place_id or data_id with the Google Maps plugin above.

Field	Type	Description
place_id	string	The place, like ChIJAyikp57iwogRFaRCSiDTihE (place_id in Google Maps results). Send this or data_id.
data_id	string	The place, like 0x88c2e29ea7a42803:0x118ad3204a42a415 (data_id in Google Maps results). Send this or place_id.
sort_by	string	most_relevant (default), newest, highest_rating or lowest_rating.
num	number	Reviews per page: 10 (default) or 20.
pages	number	How many pages in one call, 1 (default) to 10. Each page with reviews is billed.
next_page_token	string	Where to continue: search_information.next_page_token of an earlier answer. Send it with its session.
topic_id	string	Only reviews about one topic: an id from the topics list of an earlier answer.
query	string	Only reviews that mention a word, like parking. Send this or topic_id.
session	string	Keeps the calls of one place on one IP: send search_parameters.session with next_page_token.
hl	string	Language of the reviews, like en (default), de or pt-BR.

curl "https://api.crawio.com/plugin/google/maps/reviews?place_id=ChIJAyikp57iwogRFaRCSiDTihE&sort_by=newest&num=20" \
  -H "x-api-key: YOUR_API_KEY"

{
  "search_parameters": {"engine": "google_maps_reviews", "data_id": "0x88c2e29ea7a42803:0x118ad3204a42a415",
                        "place_id": "ChIJAyikp57iwogRFaRCSiDTihE", "sort_by": "newest", "num": 20, "session": "gmaps-3f1a9c0b2d4e"},
  "search_information": {"results": 20, "pages_returned": 1, "next_page_token": "CjEIARIpCgoAP7_LBz8L____Eh...:20"},
  "place_info": {"title": "City of Gulfport: Gulfport Casino Ballroom", "address": "5500 Shore Blvd S, Gulfport, FL 33707",
                 "rating": 4.6, "reviews": 834, "type": "Event venue"},
  "topics": [{"keyword": "beach", "mentions": 54, "id": "SEtKbj3r9EY"}],
  "reviews": [
    {
      "position": 1,
      "rating": 5,
      "date": "2 years ago",
      "iso_date": "2024-05-07T15:42:21Z",
      "snippet": "We love this historic waterfront venue...",
      "likes": 0,
      "images": ["https://lh3.googleusercontent.com/..."],
      "response": {"snippet": "Thank you!", "date": "2 years ago"},
      "user": {"name": "...", "local_guide": true, "reviews": 58, "photos": 137},
      "review_id": "ChdDSUhNMG9nS0VJQ0FnSURqcTZpR3NnRRAB",
      "source": "Google"
    }
  ],
  "cost": 5
}

Fields a review does not have are left out. A page that does not load is asked again on a new IP, free; if it still does not load, or your credits run out, the answer has the pages before it and search_information.stopped. To get the rest, call again for the same place with next_page_token set to stopped.next_page_token and a new Idempotency-Key. A call counts as one of your concurrent requests until it answers. A place Google does not know is 404 place_not_found, not billed.

What counts as billable

You are billed when Crawio reaches the site and the site gives a definitive answer about the resource you asked for. Everything else is free.

Code	Meaning
2xx	The site returned your content. Billable.
404 and 410	The site confirmed the resource does not exist or was removed. Billable: the request went through and the site gave a definitive answer.
403 and other 4xx	The site declined to serve this request, most often because it needs an authenticated session. Never billed.
5xx	The site's own servers failed. Never billed.
Blocked	Crawio could not return a response for the request. Never billed.

A site 4xx or 5xx is not a block. The request went through and you got the site’s own answer. The most common 403 is content behind a login: send your session cookies (see Sessions & login) and the same URL returns 200.

Why a 404 bills and a 403 does not. A 404 (or 410, gone for good) is a definitive answer: the page is gone and you can stop asking for it. A 403 is the site declining this request while the content usually still exists, so charging for it would charge you twice for one page.

How many credits. Most pages use 1 credit per billable request. Pages behind heavy protection use 6 credits. A response over 10 MB uses 5 credits, or 10 on sites served from residential IPs. A request that targets a country other than the US uses 10 credits, with the first 500 KB of the answer included and 5 credits per further 100 KB. Run a URL in the Playground to see its cost before you scale. The cost field and the X-Crawio-Cost header on every response, and the Requests tab, show what each request used.

Which number shows Crawio is working. Delivered is the share of requests that came back with the site’s real answer; Blocked is what Crawio could not deliver. If Blocked climbs, tell us.

Key limits and alerts

Give each API key its own credit limit and email alerts, so one project or environment cannot use the whole plan. Every key can have one, the Default key included.

Set a limit

On the Limits & alerts page (under API Key in the sidebar), click Add limit and choose the key. You set:

Credit limit: a whole number of credits, per billing period or per day (UTC). An account without a billing period, such as the free trial, counts per month instead.
Alert at: an email at a percentage of the limit (1 to 99, 80 by default; empty for none). The email when the limit is reached is always sent.
When reached: block requests from this key (the default), or alert only, where requests keep running and billing.

The workspace owner can set a limit on any key, and a developer on the keys they created. Only the owner can set the Default key’s limit. The Credits column on the same page shows what each key used against its limit.

When the limit is reached

The key’s requests return 429 with error key_credit_limit until the limit resets, or until you raise or remove it. Other keys keep working. Refused requests are not billed and do not appear in your request log.

HTTP/1.1 429 Too Many Requests
Content-Type: application/json
Retry-After: 1480877

{
  "error": "key_credit_limit",
  "message": "The API key staging reached its credit limit of 1 for this billing period (1 credit used). It resets when your plan renews on 28 October 2026. Other keys are not affected; raise or remove the limit on the API Key page to continue now.",
  "retry_after": 1480877,
  "key": "staging",
  "used": 1,
  "limit": 1,
  "resets_at": "2026-10-28T01:02:57Z",
  "manage": "https://crawio.com/app/#apikey"
}

Field	Type	Description
error	string	key_credit_limit.
message	string	What happened and when the limit resets, a sentence you can show or log.
key	string	The name of the key that reached its limit.
used	number	Credits the key used in this period. It can be above the limit (see Usage above the limit).
limit	number	The key's credit limit.
retry_after	number	Seconds until the limit resets. Also sent as the Retry-After header.
resets_at	string	When the limit resets, in UTC (ISO 8601).
manage	string	The API Key page, where the limit can be raised or removed.
detail	object	The same error in its older form, kept for existing integrations.

When the reset time is not known, retry_after, resets_at and the Retry-After header are left out.

Handling it in code

Do not retry a key_credit_limit before resets_at: every request on that key is refused until then. Pause the work on that key, or raise the limit. The other 429 codes mean something else:

Code	Meaning
concurrency_limit	Your plan's concurrent requests are all running. Wait Retry-After (seconds) and retry.
credits_used_up	The free trial's credits are used. They do not refill: upgrade to continue. Do not retry.
key_credit_limit	This key reached its own limit. Other keys keep working. Do not retry until resets_at.

Your plan’s own limits are checked first: while all your concurrent requests are running, or when the account is out of credits, you get that error instead.

import time, requests

r = requests.post("https://api.crawio.com/scrape", headers={"x-api-key": "YOUR_API_KEY"},
                  json={"url": "https://example.com"}, timeout=200)
if r.status_code == 429:
    err = r.json()
    if err["error"] == "concurrency_limit":
        time.sleep(float(r.headers.get("Retry-After", 2)))      # then retry
    elif err["error"] == "key_credit_limit":
        print("Key", err["key"], "is paused until", err["resets_at"])   # no retry before then

When it resets

Per billing period: when your plan renews.
Per month, on accounts without a billing period: on the 1st of the month at 00:00 UTC.
Per day: at midnight UTC.

Raising or removing the limit applies to the next request.

Usage above the limit

Requests already running when the limit is reached are completed and billed, so usage can end above the limit by up to 5 credits per concurrent request. With 50 concurrent requests that is at most 250 credits. A request with a country can add its size credits on top (see What counts as billable). The Limits & alerts page shows your account’s own figure, and every page shows the real usage, never capped at the limit.

Alerts

At a percentage: sent when the key’s usage passes it.
Limit reached: sent when usage reaches the limit, or when a request is refused for it.

Each email is sent at most once per period, or once per day for a daily limit, even if you change the limit. They go to the workspace owners and the team members picked for the key under Recipients.

Notes

Playground runs use the Default key and count toward its limit. When that key is blocked, the Playground says so and links to the API Key page.

MCP server

Connect Crawio to AI assistants and coding agents, such as Claude, Cursor and VS Code, through the Model Context Protocol (MCP). The agent can then read web pages and Google Maps data for you, with your account’s credits.

Server URL:

https://mcp.crawio.com/mcp

Connect with one click

Add the URL in your app, then sign in to Crawio and click Allow. No key to copy: the app gets its own API key, named MCP: <app>, which you can revoke any time in the API Key tab.

Claude (claude.ai and Claude Desktop): Settings, Connectors, Add custom connector. Paste the URL and click Connect.
Claude Code: run the command below, then type /mcp, pick crawio and choose Authenticate.
Cursor, VS Code, Windsurf and other apps: add the URL as a remote MCP server; the app opens the sign-in page.

claude mcp add --transport http crawio https://mcp.crawio.com/mcp

Connect with an API key

For apps that take a header instead of a sign-in, send your key as Authorization: Bearer YOUR_API_KEY.

claude mcp add --transport http crawio https://mcp.crawio.com/mcp \
  --header "Authorization: Bearer YOUR_API_KEY"

Tools

Tool	Cost	What it does
scrape_page	as /scrape	Any public page as clean Markdown (the main content), plain text, HTML or the list of its links. Takes a country and a session_id like /scrape.
google_maps_search	5 per page	Places as data: name, rating, reviews, category, address, phone, website, coordinates.
google_maps_reviews	5 per page	A place's reviews, sorted or filtered by a word, with the owner's responses.
get_account	free	Your plan, credits left and renewal date.

Long pages come in parts. scrape_page returns up to 20,000 characters and says where the next part starts. Reading the rest of the same page within 10 minutes uses no credits, so a long page never fills the agent’s context at once.

Try asking

“Read https://example.com/pricing and list the plans and prices.”
“Show me this page the way a visitor from Germany sees it.”
“Find dentists in Austin, TX with a rating of 4.5 or more, with their websites.”
“Summarize the 40 newest reviews of the first result.”
“How many Crawio credits do I have left?”

Good to know

Billing. The same as the API: every answer shows the credits it used, and the requests appear in your request log.
Read only. The tools read pages and data; they never send forms or change anything on a site.
Remove access. Revoke the app’s MCP: <app> key in the API Key tab; the app stops at its next request.

Scrapy

Drop-in downloader middleware: matched domains route through Crawio; spiders stay normal. The package is scrapy-crawio on PyPI.

pip install scrapy-crawio