Sitemap

`POST https://api.keirolabs.cloud/api/v2/sitemap`

Cost: 1 credit · Latency: ~1–5 s

Every URL a site publishes: reads robots.txt, follows sitemap indexes (and .gz files), and returns each page URL with its lastmod. Filter by path or by date.

**When to use it.** Use Sitemap to list every page on a site before extracting: docs crawls, competitor page monitoring, finding new blog posts by date.

**When not to.** Don't use Sitemap for page content (use Extract) or for sites that publish no sitemap.

## Request

**cURL**

```bash
curl -X POST https://api.keirolabs.cloud/api/v2/sitemap \
  -H "Authorization: Bearer $KEIRO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "stripe.com", "path_prefix": "/blog", "limit": 500}'
```

**Python**

```python
import os, requests

res = requests.post(
    "https://api.keirolabs.cloud/api/v2/sitemap",
    headers={"Authorization": f"Bearer {os.environ['KEIRO_API_KEY']}"},
    json={
        "url": "stripe.com",
        "path_prefix": "/blog",
        "limit": 500
    },
)
print(res.json())
```

**Node.js**

```js
const res = await fetch("https://api.keirolabs.cloud/api/v2/sitemap", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.KEIRO_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    "url": "stripe.com",
    "path_prefix": "/blog",
    "limit": 500
  }),
});
console.log(await res.json());
```

## Parameters

| Name | Type | Required | Description |
| --- | --- | --- | --- |
| `url` | string | yes | Domain or site URL, e.g. "stripe.com". |
| `path_prefix` | string | no | Only return URLs whose path starts with this, e.g. "/blog". |
| `since` | string | no | ISO date. Only URLs with a lastmod on or after it (URLs without lastmod are skipped and counted). |
| `limit` | integer | no | Max URLs returned. Default and max: 50000. |

## Response

```json title="200 OK"
{
  "site": "https://stripe.com",
  "source": "robots.txt",
  "sitemaps": [
    "https://stripe.com/sitemap/sitemap.xml"
  ],
  "total_urls": 2,
  "truncated": false,
  "urls": [
    {
      "url": "https://stripe.com/pricing",
      "lastmod": "2026-10-01"
    },
    {
      "url": "https://stripe.com/blog",
      "lastmod": null
    }
  ],
  "creditsCharged": 1
}
```

| Name | Type | Required | Description |
| --- | --- | --- | --- |
| `urls` | array | no | Page URLs with lastmod (null when the sitemap has none). |
| `sitemaps` | array | no | Sitemap files that were read. |
| `source` | string | no | "robots.txt" if declared there, else "common paths". |
| `truncated` | boolean | no | True when limit or the 60-file cap was hit. |
| `skipped_without_lastmod` | integer | no | Only with since: URLs dropped because they carry no lastmod. |

## Errors

| Status | Code | Cause | Fix |
| --- | --- | --- | --- |
| `400` | `VALIDATION_ERROR` | Missing url or bad path_prefix/since/limit | Send a domain; path_prefix must start with /. |
| `402` | `INSUFFICIENT_CREDITS` | Not enough credits | Buy more credits or upgrade your plan at /dashboard/billing. |
| `400` | `NO_SITEMAP` | No sitemap in robots.txt or common paths | Not charged. Use Search or Extract to discover pages. |

## Rate limits

| Tier | Limit |
| --- | --- |
| Free | 5/min |
| Starter | 20/min |
| Pro | 40/min |
| Startup | 80/min |
| Enterprise | 120/min |

Over the limit you get a `429`. See [Rate limits](/rate-limits) for headers and backoff.