Scraping: a whole catalog

Walk pagination in one call — up to 50 pages per request. You pay only for the pages actually returned, not for the attempt.

Overview

Most sites split a catalog across many pages. Parsing them one by one costs you a call, a charge and a log entry per page — and if page 14 breaks you still pay for 13 pages and get nothing usable.

parse_pages does the whole walk server-side: it follows pagination, parses every page the same way parse_page does, and returns one response with all pages in items[].

How the next page is found — three ways, in order of reliability:

1. rel="next" link in the markup (standard);
2. link text: next, more, ›, », →;
3. URL pattern: ?page=N, &page=N, ?p=, ?pg=, /page/N.

If none of them matches, we stop and say so (stop_reason: "no_next"). We never guess URLs and never hammer a site guessing at page numbers.

curl -X POST https://api.serp-base.com/v1/query \
  -H "Content-Type: application/json" \
  -d '{"tool": "parse_pages", "query": "https://shop.example/catalog", "limit": 10}'

Parameters

Code examples

cURL

curl -X POST https://api.serp-base.com/v1/query \
  -H "Content-Type: application/json" \
  -d '{"tool": "parse_pages", "query": "https://shop.example/catalog", "limit": 5}'

Python

import requests

r = requests.post("https://api.serp-base.com/v1/query", json={
    "tool": "parse_pages",
    "query": "https://shop.example/catalog",
    "limit": 5,
})
data = r.json()["data"]
print(data["pages_fetched"], "pages,", data["stop_reason"])
for page in data["items"]:
    print(page["page"], page["title"], page["length"])

JavaScript

const r = await fetch('https://api.serp-base.com/v1/query', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    tool: 'parse_pages',
    query: 'https://shop.example/catalog',
    limit: 5,
  }),
});
const { data } = await r.json();
console.log(data.pages_fetched, data.stop_reason);

PHP

$ch = curl_init('https://api.serp-base.com/v1/query');
curl_setopt_array($ch, [
  CURLOPT_POST => true,
  CURLOPT_HTTPHEADER => ['Content-Type: application/json'],
  CURLOPT_POSTFIELDS => json_encode([
    'tool' => 'parse_pages',
    'query' => 'https://shop.example/catalog',
    'limit' => 5,
  ]),
  CURLOPT_RETURNTRANSFER => true,
]);
$data = json_decode(curl_exec($ch), true)['data'];

Response

{
  "ok": true,
  "data": {
    "url": "https://shop.example/catalog",
    "format": "text",
    "pages_requested": 5,
    "pages_fetched": 4,
    "pages_failed": 0,
    "stop_reason": "no_next",
    "next_method": "rel_next",
    "items": [
      { "page": 1, "url": "https://shop.example/catalog",     "title": "Catalog", "length": 18240, "content": "…" },
      { "page": 2, "url": "https://shop.example/catalog?p=2", "title": "Catalog", "length": 17902, "content": "…" }
    ]
  }
}

stop_reason — why the walk ended:

• limit — reached the requested number of pages;
• no_next — no next page found (end of catalog);
• duplicate — the site returned the same content again (guards against fake pagination);
• fetch_error — a page failed to load; the walk stops there.

next_method — which of the three detection methods worked (rel_next, query:page, path:/page/N), or null.

Payment

Pay per page with USDC on Base via the x402 protocol. Gas is paid by the facilitator — you only need USDC in your wallet.

• Network: Base mainnet (eip155:8453)
• Asset: USDC
• Price: $2.00 per 1,000 pages — that is $0.002 per page
• Charged per page actually fetched, not per page requested

Ask for limit: 10, get 3 pages — you pay for 3. Get none — you pay nothing. The answer always carries pages_charged so the math is never a surprise.

No payment is taken when a service fails. We run the work first and settle afterwards.

Errors

Related tools

Scraping: URL to data · Scraping with stealth · Extract data by schema