The CrawlNode API, from first call to production

Quickstart

Three steps from nothing to a screenshot on your disk: get a key, make four calls, check what came back.

1. Get a key

1. Create an account and confirm your email address.
2. On the API keys page, choose how many physical nodes you want and send the request. We confirm the order with you, usually within one business day, and the key appears on that page.
3. Reveal the key. It is shown once, with these same calls beside it and the key already filled in.

Then put it in your environment, which is where every sample here reads it from:

export CRAWLNODE_TOKEN='paste-your-key-here'

2. Make the four calls

The base URL is https://api1.crawlnode.com. Start a browser, navigate, take a screenshot, destroy.

# 1. start a browser (warm pool, managed proxy)
SESSION=$(curl -s -X POST https://api1.crawlnode.com/api/start \
  -H "Token: $CRAWLNODE_TOKEN" -H "Content-Type: application/json" \
  -d '{"proxy":"auto"}' | sed -n 's/.*"session_id":"\([^"]*\)".*/\1/p')

# 2. navigate
curl -s -X POST https://api1.crawlnode.com/api/go \
  -H "Token: $CRAWLNODE_TOKEN" -H "X-Session-Id: $SESSION" \
  -H "Content-Type: application/json" -d '{"url":"https://example.com"}'

# 3. screenshot of the page (WebP bytes); drop "area" for the whole window
curl -s -X POST https://api1.crawlnode.com/api/screenshot \
  -H "Token: $CRAWLNODE_TOKEN" -H "X-Session-Id: $SESSION" \
  -H "Content-Type: application/json" -d '{"area":"viewport"}' -o page.webp

# 4. always destroy
curl -s -X POST https://api1.crawlnode.com/api/destroy \
  -H "Token: $CRAWLNODE_TOKEN" -H "X-Session-Id: $SESSION" \
  -H "Content-Type: application/json" -d '{}'

import json, os, urllib.request

BASE = "https://api1.crawlnode.com"
TOKEN = os.environ["CRAWLNODE_TOKEN"]

def call(path, body, session=None, raw=False):
    headers = {"Token": TOKEN, "Content-Type": "application/json"}
    if session:
        headers["X-Session-Id"] = session
    req = urllib.request.Request(BASE + path, json.dumps(body).encode(), headers)
    with urllib.request.urlopen(req, timeout=90) as res:
        data = res.read()
    return data if raw else json.loads(data)

session = call("/api/start", {"proxy": "auto"})["session_id"]
try:
    call("/api/go", {"url": "https://example.com"}, session)
    with open("page.webp", "wb") as f:  # the page alone; {} captures the whole window
        f.write(call("/api/screenshot", {"area": "viewport"}, session, raw=True))
finally:
    call("/api/destroy", {}, session)

import { writeFile } from "node:fs/promises";

const BASE = "https://api1.crawlnode.com";
const TOKEN = process.env.CRAWLNODE_TOKEN;

async function call(path, body, session) {
  const headers = { Token: TOKEN, "Content-Type": "application/json" };
  if (session) headers["X-Session-Id"] = session;
  const res = await fetch(BASE + path, { method: "POST", headers, body: JSON.stringify(body) });
  if (!res.ok) throw new Error(`${path} -> HTTP ${res.status}`);
  return res;
}

const { session_id } = await (await call("/api/start", { proxy: "auto" })).json();
try {
  await call("/api/go", { url: "https://example.com" }, session_id);
  const shot = await call("/api/screenshot", { area: "viewport" }, session_id); // {} = whole window
  await writeFile("page.webp", Buffer.from(await shot.arrayBuffer()));
} finally {
  await call("/api/destroy", {}, session_id);
}

The session_id comes back both in the body and in the X-Session-Id response header.

3. Check it worked

You now have page.webp in the directory you ran the calls from: the page as the browser painted it. If a call failed instead, its body says why, and Errors and retries says what to do about each status.

To check a key on its own, without starting a browser, ask what it has used. This is the one call that needs no session:

curl -s -X POST https://api1.crawlnode.com/api/usage \
  -H "Token: $CRAWLNODE_TOKEN" -H "Content-Type: application/json" -d '{}'

It answers with the limits on the key and what has been spent against them this period. A 401 here means the key itself is wrong, before you have spent any time on a session.

Endpoints

All are POST /api/<name>. The ones marked session need X-Session-Id.

Proxies and targeting

Every session leaves through an exit you choose on start. Managed exits are part of every node and are in the United States today. A proxy of your own is used as given and never counted, and is the way to arrive from anywhere else.

A residential exit, for a site that blocks hosting ranges:

SESSION=$(curl -s -X POST https://api1.crawlnode.com/api/start \
  -H "Token: $CRAWLNODE_TOKEN" -H "Content-Type: application/json" \
  -d '{"proxy":"rotating_residential"}' \
  | sed -n 's/.*"session_id":"\([^"]*\)".*/\1/p')

Then ask the session where it is leaving from, before you trust what the site shows you. egress answers with the address the site sees:

curl -s -X POST https://api1.crawlnode.com/api/egress \
  -H "Token: $CRAWLNODE_TOKEN" -H "X-Session-Id: $SESSION" \
  -H "Content-Type: application/json" -d '{}'

proxy on a live session switches the exit without restarting Chrome, so one browser can look at the same page from two addresses.

Errors and retries

Success is 200. Errors carry a JSON body: {"error": {"code", "message"}} from the dispatcher, or {"detail": "..."} from the browser agent. Branch on the code, not the text.

• 401 unknown session or invalid token. Start a new session.
• 429 rate limit exceeded. Wait for Retry-After seconds and retry.
• 503 no free node, or no proxy passed the probe. Also carries Retry-After; retry after it.
• 504 the agent timed out. Retry once; the session is still valid.
• 4xx anything else is yours to fix and will fail identically on a retry.

Good practice

• Always destroy, including on failure. A leaked session holds a Chrome and a node slot until the idle timer reclaims it after 30 minutes.
• Send JSON objects, never arrays. An empty body is {}.
• Use wait instead of polling view. A view walks the whole accessibility tree; wait answers the same question without shipping the page.
• Read X-Screenshot-Settled. It is present with value 0 only when the page never painted. Absent is the good case.
• Prefer script for multi-step work. One round trip instead of one per step, and a failing step does not fail the run.