Website Crawler API

Free Public API

Recursively crawl entire websites, discover URLs, and build site hierarchies with smart boundaries.

Try in Live Web Tool

POSThttps://api.headless-lender.com/api/v1/crawl

Public: No Key Needed•Highest Free Quota: 1,000 Req/Mo + 10h Browsers (No Card Required)

Overview & Architecture

Map domain hierarchies, index blog posts, and discover deep URLs with configurable maximum depth, page limits, path prefix filters, and loopback security controls.

Configurable crawl depth and page limits

Strict domain isolation and prefix filtering

SSRF private network boundary enforcement

Full title and status code extraction per page

Playwright Execution & Valkey Caching Pipeline

Because this endpoint utilizes full headless Chromium rendering, dynamic JavaScript and single-page apps (SPAs) are executed completely before conversion:

1. Cold Scrape (Cache MISS)~800ms – 2.5s

Chromium initializes, fetches the URL, executes client JavaScript, resolves DOM mutations, and generates output.

2. Repeat Request (Cache HIT)~5ms – 15ms

Returned instantly from Valkey/Redis cache with header X-Lender-Cache: HIT (15-minute TTL).

Code Integration Examples

Instant Public Endpoint: Zero configuration, no API key needed. Want our highest free quota? Get 1,000 free requests/month + 10 hours of cloud browsers with a free key (No credit card required).

curl -X POST https://api.headless-lender.com/api/v1/crawl \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://news.ycombinator.com",
    "maxPages": 10,
    "maxDepth": 2,
    "sameDomainOnly": true
  }'

Headers & Authentication Standard

1. Public Access (No Key)

No Authorization header required. Simply send a JSON POST request. Perfect for quick prototypes, small scripts, and low-volume scraping.

Content-Type: application/json

2. Free API Key (Highest Free Quota)

1,000 Req/Mo Free

Sign up in 10s with no credit card required. Unlocks 1,000 free monthly requests, 600 browser minutes (10 hours), 120 req/min burst limits, and cache-bypass controls.

Authorization: Bearer hl_live_your_api_key_here

Access Mode	Monthly Quota	Cloud Browser Time	Burst Limit	Credit Card	Action
Public Anonymous	Daily IP Allowance	5-min trial sandbox	10 req/min	None	Instant (No Setup)
Free Account (Highest Quota)	1,000 Req / Month	600 Mins (10 Hours)	120 req/min	Zero ($0 forever)	Claim Free Key →

Request Body Parameters

Field	Type	Required	Default	Description
url	string	required	—	Root seed URL to begin crawling.
maxPages	number	optional	10	Maximum number of pages to discover and visit.
maxDepth	number	optional	2	Maximum hyperlink hop depth from seed URL.
sameDomainOnly	boolean	optional	true	Stay strictly within the origin domain.
allowedPathPrefixes	string[]	optional	—	Only follow links starting with these path prefixes (e.g. ['/docs', '/blog']).

Response Schema & Example

200 OK — application/jsonDuration: ~800ms (Cold) / ~8ms (Cached)

{
  "seedUrl": "https://news.ycombinator.com",
  "pagesCrawled": 5,
  "pages": [
    {
      "url": "https://news.ycombinator.com",
      "title": "Hacker News",
      "depth": 0,
      "status": 200
    },
    {
      "url": "https://news.ycombinator.com/newest",
      "title": "New Links | Hacker News",
      "depth": 1,
      "status": 200
    },
    {
      "url": "https://news.ycombinator.com/front",
      "title": "Past Front Pages",
      "depth": 1,
      "status": 200
    }
  ],
  "durationMs": 1450
}

Webpage Translator Table Extractor