> ## Documentation Index
> Fetch the complete documentation index at: https://usefoil.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# API abuse & scraping

> Stop API abuse and LLM scraping while allowing legitimate crawlers. Verify Foil sessions across API calls and split policy by attribution.

> [!NOTE]
>
> APIs are different from form-submit surfaces. A single page can make many API calls, so you decide how often to ask Foil for a fresh handoff. And you want to distinguish LLM scrapers (`ai-agent`) from crawlers and signed agents that you allow, because the right answer for those is different.

## The threat

Two problems get lumped together as "API abuse," and they want different responses:

* **LLM scraping.** Cloud-hosted browser agents and their self-hosted equivalents scrape the web on behalf of model training, retrieval pipelines, or end-user automation. They look like browsers because they *are* browsers, just driven by an agent loop. Foil attributes these as `ai-agent`.
* **Scripted scraping.** Classic headless Chrome iterating your listing endpoints, pricing APIs, or search results. Foil attributes these as `automation`. HTTP-level clients that don't run a browser never produce a Foil session, so the guard below rejects their requests because they carry no token.

Against both, you want something stronger than a per-user-account rate limit - attackers rotate accounts - and stronger than a per-IP rate limit - attackers rotate residential proxies. The visitor fingerprint is the axis that holds.

At the same time, you almost certainly *do* want some non-human traffic:

* **Search crawlers** (Googlebot, Bingbot) - you want them indexing your public content. Foil labels a session `crawler` from the request's User-Agent header, which anyone can send, so verify a crawler before you trust it.
* **Signed agents** - some browser agents sign their requests with HTTP message signatures (Web Bot Auth) to identify themselves. When Foil verifies a signature, it adds a `trust` label with the value `verified` to the session's attribution, and it adds the signing domain as a `product` label. For signers that Foil recognizes, the actor label is `ai-agent`, so a policy that blocks every `ai-agent` session also blocks these agents.
* **Your own internal tooling** - health checks, monitoring, analytics pipelines. These calls don't come from a browser, so they have no Foil session. Authenticate them with API keys.

The integration on this page separates "block" from "allow" on the attribution, not just the verdict. In the sealed token, `attribution.bot.labels` lists labels that each have a `kind`, a `value`, and a `confidence` from `0` to `1`. The actor label is the one whose `kind` is `actor`, and its value is `automation`, `ai-agent`, `crawler`, or `unknown`. A session can have no actor label, so treat a missing label as `unknown`.

## The flow

**1. Start Foil once at app boot**

For authenticated SPAs, start the client when the shell mounts. For public APIs accessed directly (no browser), see the bottom of this page.

**2. Request a handoff when the page makes a protected call**

Call `getSession()` and send the `sealedToken` as a header. Every call returns a fresh token for the same `sessionId`. To avoid a Foil call for every request on a chatty page, you can reuse a handoff for a short time.

**3. Verify on each protected request**

Verify the token locally with `safeVerifyFoilToken()` and reject a token that is more than a couple of minutes old. For very hot endpoints, look up the verified session in the Sessions API instead and cache the verdict.

**4. Re-verify on high-value actions or periodically**

For mutations or sensitive reads, request a fresh handoff. For cached verdicts, refresh every few minutes.

**5. Split policy by attribution**

Allow signed agents (`trust` label `verified`) and verified crawlers on read requests; block `automation` and `ai-agent`; treat `human` normally.

## Client integration

Request a sealed handoff when the page makes a protected API call, and send the token as a header. `getSession()` is a call to Foil, so it adds latency to the request that waits for it. The example reuses a handoff for up to one minute so that a burst of requests doesn't make a Foil call each time, and it requests a fresh handoff for high-value actions.

A token that the page reuses is a token that a script could capture from a real browser session and replay. Keep the reuse window short, and set the maximum token age on the server only slightly above it. The example reuses a handoff for one minute, and the server in the next section accepts tokens that are up to two minutes old.

```html theme={"dark"}
<script type="module">
  const foilPromise = import("https://cdn.usefoil.com/t.js").then(
    (Foil) =>
      Foil.start({
        publishableKey: "pk_live_your_publishable_key",
      }),
  );

  const HANDOFF_REUSE_MS = 60 * 1000;
  let cached = null;

  async function getFoilHandoff({ fresh = false } = {}) {
    if (!fresh && cached && Date.now() - cached.at < HANDOFF_REUSE_MS) {
      return cached.handoff;
    }

    try {
      const foil = await foilPromise;
      const handoff = await foil.getSession();
      cached = { handoff, at: Date.now() };
      return handoff;
    } catch (err) {
      // Send the request without a handoff. The server rejects it.
      return null;
    }
  }

  async function apiFetch(path, init = {}, { fresh = false } = {}) {
    const handoff = await getFoilHandoff({ fresh });
    const headers = new Headers(init.headers);
    if (handoff) {
      headers.set("X-Foil-Token", handoff.sealedToken);
    }
    return fetch(path, { ...init, headers });
  }

  // Request a fresh handoff for high-value mutations
  function apiMutate(path, init) {
    return apiFetch(path, init, { fresh: true });
  }
</script>
```

If the script fails to load or `getSession()` fails, the request goes out without a token and the guard in the next section rejects it. If you also serve clients that don't use your front end, route them by API key before the guard.

## Server verification: two patterns

You have two ways to check a session on your server. You can verify the sealed token on each protected request, which is a local crypto operation with no network call, or you can bind the verified session to your own session and read its latest verdict from the Sessions API.

The server SDKs verify a token's contents but don't check its age. Both patterns need your own age check on `decision.evaluated_at`, so that a token captured from one session can't be replayed later.

### Pattern A: verify the sealed token per request

The examples do the following:

1. Verify the sealed token and treat it as missing if verification fails or if the token is more than two minutes old.
2. Allow a signed agent (a session with a `trust` label of `verified`) on `GET` requests, and block it on everything else.
3. Allow a session that Foil labels `crawler` on `GET` requests only after your own check confirms the crawler. The verified-crawler helper in each example stands in for that check. Use a reverse DNS lookup of the request's IP address that a forward lookup confirms, or compare the address with the IP ranges that the crawler's operator publishes.
4. Block every other session with a `bot` verdict.

```javascript Node.js theme={"dark"}
const { safeVerifyFoilToken } = require("@abxy/foil-server");

const MAX_TOKEN_AGE_MS = 2 * 60 * 1000;

function verifyFoil(sealedToken) {
  if (!sealedToken) return null;

  const result = safeVerifyFoilToken(sealedToken, process.env.FOIL_SECRET_KEY);
  if (!result.ok) return null;

  const age = Date.now() - Date.parse(result.data.decision.evaluated_at);
  if (age > MAX_TOKEN_AGE_MS) return null;

  return result.data;
}

async function foilGuard(req, res, next) {
  const foil = verifyFoil(req.get("X-Foil-Token"));
  if (!foil) {
    return res.status(401).json({ error: "Missing or invalid Foil token" });
  }

  const labels = foil.attribution?.bot?.labels ?? [];
  const actor = labels.find((label) => label.kind === "actor")?.value ?? null;
  const isSignedAgent = labels.some(
    (label) => label.kind === "trust" && label.value === "verified",
  );

  // A verified signature identifies the agent. Allow it on reads only.
  if (isSignedAgent) {
    if (req.method !== "GET") return res.status(403).json({ error: "Blocked" });
    req.foilVerdict = { verdict: "signed_agent", actor };
    return next();
  }

  // The crawler label comes from the User-Agent header, which anyone can send.
  // Allow it only after you verify the crawler.
  if (req.method === "GET" && actor === "crawler" && (await isVerifiedCrawler(req))) {
    req.foilVerdict = { verdict: "verified_crawler", actor };
    return next();
  }

  // Block automation, ai-agent, and other bots regardless of method
  if (foil.decision.verdict === "bot") {
    return res.status(403).json({ error: "Blocked" });
  }

  req.foilVerdict = { verdict: foil.decision.verdict, actor };
  next();
}

app.use("/api", foilGuard);
```

```python Python theme={"dark"}
import os
from datetime import datetime, timezone

from foil_server import safe_verify_foil_token

MAX_TOKEN_AGE_SECONDS = 2 * 60


def verify_foil(sealed):
    if not sealed:
        return None

    result = safe_verify_foil_token(sealed, os.environ["FOIL_SECRET_KEY"])
    if not result.ok:
        return None

    evaluated_at = datetime.fromisoformat(
        result.data.decision.evaluated_at.replace("Z", "+00:00")
    )
    age = (datetime.now(timezone.utc) - evaluated_at).total_seconds()
    if age > MAX_TOKEN_AGE_SECONDS:
        return None

    return result.data


def foil_guard(request):
    foil = verify_foil(request.headers.get("X-Foil-Token"))
    if foil is None:
        return {"error": "Missing or invalid Foil token"}, 401

    labels = (foil.attribution.bot or {}).get("labels", [])
    actor = next((label["value"] for label in labels if label["kind"] == "actor"), None)
    is_signed_agent = any(
        label["kind"] == "trust" and label["value"] == "verified" for label in labels
    )

    # A verified signature identifies the agent. Allow it on reads only.
    if is_signed_agent:
        if request.method != "GET":
            return {"error": "Blocked"}, 403
        request.state.foil = {"verdict": "signed_agent", "actor": actor}
        return None

    # The crawler label comes from the User-Agent header, which anyone can send.
    # Allow it only after you verify the crawler.
    if request.method == "GET" and actor == "crawler" and is_verified_crawler(request):
        request.state.foil = {"verdict": "verified_crawler", "actor": actor}
        return None

    if foil.decision.verdict == "bot":
        return {"error": "Blocked"}, 403

    request.state.foil = {"verdict": foil.decision.verdict, "actor": actor}
    return None
```

```go Go theme={"dark"}
import (
    "net/http"
    "os"
    "time"

    foil "github.com/abxy-labs/foil-server-go"
)

const maxTokenAge = 2 * time.Minute

// tokenActor returns the actor label, and whether the token carries a
// trust label with the value "verified".
func tokenActor(token *foil.VerifiedFoilToken) (actor string, signed bool) {
    labels, _ := token.Attribution.Bot["labels"].([]any)
    for _, item := range labels {
        label, ok := item.(map[string]any)
        if !ok {
            continue
        }
        kind, _ := label["kind"].(string)
        value, _ := label["value"].(string)
        switch {
        case kind == "actor":
            actor = value
        case kind == "trust" && value == "verified":
            signed = true
        }
    }
    return actor, signed
}

func FoilGuard(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        tr := foil.SafeVerifyFoilToken(
            r.Header.Get("X-Foil-Token"),
            os.Getenv("FOIL_SECRET_KEY"),
        )
        if !tr.OK {
            http.Error(w, "Missing or invalid Foil token", 401)
            return
        }

        evaluatedAt, err := time.Parse(time.RFC3339, tr.Data.Decision.EvaluatedAt)
        if err != nil || time.Since(evaluatedAt) > maxTokenAge {
            http.Error(w, "Missing or invalid Foil token", 401)
            return
        }

        actor, signed := tokenActor(tr.Data)

        // A verified signature identifies the agent. Allow it on reads only.
        if signed {
            if r.Method != http.MethodGet {
                http.Error(w, "Blocked", 403)
                return
            }
            next.ServeHTTP(w, r)
            return
        }

        // The crawler label comes from the User-Agent header, which anyone
        // can send. Allow it only after you verify the crawler.
        if r.Method == http.MethodGet && actor == "crawler" && isVerifiedCrawler(r) {
            next.ServeHTTP(w, r)
            return
        }

        if tr.Data.Decision.Verdict == "bot" {
            http.Error(w, "Blocked", 403)
            return
        }

        next.ServeHTTP(w, r)
    })
}
```

```ruby Ruby theme={"dark"}
require "foil/server"
require "time"

MAX_TOKEN_AGE_SECONDS = 2 * 60

before "/api/*" do
  sealed = request.env["HTTP_X_FOIL_TOKEN"]
  halt 401, { error: "Missing Foil token" }.to_json unless sealed

  tr = Foil::Server::SealedToken.safe_verify_foil_token(sealed, ENV.fetch("FOIL_SECRET_KEY"))
  halt 401, { error: "Invalid Foil token" }.to_json unless tr[:ok]

  token = tr[:data]
  age = Time.now - Time.iso8601(token[:decision][:evaluated_at])
  halt 401, { error: "Invalid Foil token" }.to_json if age > MAX_TOKEN_AGE_SECONDS

  labels = token.dig(:attribution, :bot, :labels) || []
  actor = labels.find { |label| label[:kind] == "actor" }&.dig(:value)
  signed_agent = labels.any? { |label| label[:kind] == "trust" && label[:value] == "verified" }

  # A verified signature identifies the agent. Allow it on reads only.
  if signed_agent
    halt 403, { error: "Blocked" }.to_json unless request.get?
    next
  end

  # The crawler label comes from the User-Agent header, which anyone can send.
  # Allow it only after you verify the crawler.
  next if request.get? && actor == "crawler" && verified_crawler?(request)

  halt 403, { error: "Blocked" }.to_json if token[:decision][:verdict] == "bot"
end
```

```php PHP theme={"dark"}
use Foil\Server\SealedToken;

const MAX_TOKEN_AGE_SECONDS = 2 * 60;

function foilGuard(): void {
    $sealed = $_SERVER["HTTP_X_FOIL_TOKEN"] ?? null;
    if (!$sealed) { http_response_code(401); exit; }

    $tr = SealedToken::safeVerify($sealed, getenv("FOIL_SECRET_KEY"));
    if (!$tr->ok) { http_response_code(401); exit; }

    $age = time() - strtotime($tr->data->decision["evaluated_at"]);
    if ($age > MAX_TOKEN_AGE_SECONDS) { http_response_code(401); exit; }

    $actor = null;
    $isSignedAgent = false;
    foreach ($tr->data->attribution["bot"]["labels"] ?? [] as $label) {
        if ($label["kind"] === "actor") {
            $actor = $label["value"];
        }
        if ($label["kind"] === "trust" && $label["value"] === "verified") {
            $isSignedAgent = true;
        }
    }
    $isRead = $_SERVER["REQUEST_METHOD"] === "GET";

    // A verified signature identifies the agent. Allow it on reads only.
    if ($isSignedAgent) {
        if (!$isRead) { http_response_code(403); exit; }
        return;
    }

    // The crawler label comes from the User-Agent header, which anyone can send.
    // Allow it only after you verify the crawler.
    if ($isRead && $actor === "crawler" && isVerifiedCrawler()) {
        return;
    }

    if ($tr->data->decision["verdict"] === "bot") {
        http_response_code(403);
        exit;
    }
}
```

```bash cURL theme={"dark"}
# Session readback for the Sessions API pattern (see below)
curl https://api.usefoil.com/v1/sessions/sid_... \
  -H "Authorization: Bearer sk_live_..."
```

### Pattern B: read the verdict from the Sessions API

For very hot endpoints, verify a fresh token once, store the verified `session_id` in your own server-side session, and read the session's latest verdict from Foil on later requests, caching it for a few minutes. Foil's server-side behavioral scoring can move a verdict between the `snapshot` and `behavioral` phases as evidence accumulates, so a lookup can be more current than the token that you verified first.

Take the session ID from a token that you verified, not from a request header. A client can send any session ID, including one that it copied from a real browser session.

```javascript Node.js theme={"dark"}
const { Foil, FoilApiError } = require("@abxy/foil-server");
const client = new Foil({ secretKey: process.env.FOIL_SECRET_KEY });
const verdictCache = new Map(); // session_id → { verdict, actor, fetchedAt }

const CACHE_MS = 5 * 60_000; // refresh after 5 minutes
const MAX_STALE_MS = 15 * 60_000; // never serve a verdict older than 15 minutes

// `sessionId` is the `session_id` of a token that you verified and stored in
// your own server-side session.
async function getCachedVerdict(sessionId) {
  const cached = verdictCache.get(sessionId);
  const age = cached ? Date.now() - cached.fetchedAt : Infinity;
  if (age < CACHE_MS) return cached;

  let session;
  try {
    session = await client.sessions.get(sessionId);
  } catch (err) {
    // Foil limits how many requests your secret keys can make per minute.
    // When it does, keep using the last verdict for a limited time.
    if (err instanceof FoilApiError && err.status === 429 && age < MAX_STALE_MS) {
      return cached;
    }
    throw err;
  }

  const actor =
    session.attribution?.labels.find((label) => label.kind === "actor")?.value ?? null;
  const entry = {
    verdict: session.decision.automation_status, // "automated" | "human" | "uncertain"
    actor,
    fetchedAt: Date.now(),
  };
  verdictCache.set(sessionId, entry);
  return entry;
}
```

The session readback returns `decision.automation_status` (`"automated" | "human" | "uncertain"`) rather than the sealed-token `verdict` field, and its `attribution` is `null` when Foil has nothing to attribute. The session's `attribution.labels` have the same `kind` and `value` as the token's labels, plus a display `label` and a `confidence` from `0` to `100`. See [Server verification](https://usefoil.com/docs/server-verification.md) for the full shape.

Foil limits all of your organization's secret keys together to 600 requests per minute by default, and a limited request returns `429` with a `Retry-After` header. See [Rate limits](https://usefoil.com/docs/api-reference/rate-limits.md). The server SDKs don't retry for you, so the example keeps using the last cached verdict when Foil returns a `429`, but only until that verdict is 15 minutes old. After that, `getCachedVerdict()` throws, and your handler should treat the session as having no verdict and ask the client for a fresh token. Each cached session costs one request per cache period, so a longer period lets you serve more concurrent sessions within the limit.

## When to re-verify

Session reuse is great for throughput, but a long-lived session becomes a stale verdict. Three refresh triggers worth coding:

* **High-value mutations** - a user changing their email, deleting data, exporting an archive. Always ask for a fresh sealed handoff from the client, as `apiMutate` does in the client example.
* **Periodic refresh** - every 5–15 minutes for long sessions, re-fetch `GET /v1/sessions/:sessionId`. Foil's server-side behavioral scoring can move a verdict between `snapshot` and `behavioral` phases as evidence accumulates, and you want the latest. Each refresh counts against your secret-key rate limit, as described in Pattern B.
* **Suspicious pattern observed** - your own application logic (burst of identical queries, geographic jump) can trigger a client-side `foil.getSession()` that produces a new, freshly-attested handoff.

## Splitting policy by attribution

For read APIs, the policy matrix that works on most sites:

| Session | GET | POST / PUT / DELETE |
| - | - | - |
| `human` verdict | Allow | Allow |
| Signed agent (`trust` label `verified`) | Allow | Block (agents generally shouldn't mutate) |
| `crawler` that you verified | Allow | Block |
| `crawler` that you couldn't verify | Treat as `bot`: block or throttle | Block |
| `automation` | Block or throttle | Block |
| `ai-agent` without a `verified` trust label | Block or throttle | Block |
| `unknown` or no actor label, with a `bot` verdict | Throttle | Block |

Check the `trust` label before you check the actor label. Foil gives signed agents from signers it recognizes the actor label `ai-agent`, so a policy that blocks every `ai-agent` session also blocks the agents that identify themselves.

"Allow" doesn't mean unlimited. Keep a generous rate limit on signed agents and verified crawlers - a badly-written crawler can still hurt you - but don't return 403s.

If you want AI agents to read your content but not scrape it, set different caps for `ai-agent`: a low QPS limit that's fine for an agent answering one user's question and painful for a training-data crawler.

## APIs called directly (no browser)

Some API consumers don't run a browser at all - a mobile app, a server-to-server integration, a CLI tool. Requests to your own API go to your servers, not to Foil, so Foil never sees them. Direct API traffic has no Foil session, and the browser SDK doesn't apply. For this traffic:

* **Require API keys.** Issue a key to each customer or integration, and route requests that don't carry a Foil token down the API-key path. Apply Foil only to requests from your browser front end.
* **Rate limit by key.** Limit each API key, and limit by IP address the traffic that has no key. A per-key limit contains a leaked or shared key.
* **Use the native SDKs for your own mobile apps.** The [iOS SDK](https://usefoil.com/docs/ios-sdk.md) and the [Android SDK](https://usefoil.com/docs/android-sdk.md) return a sealed handoff that you verify on your server in the same way as a browser token.

## What's next

> [card] [User-generated content](https://usefoil.com/docs/use-cases/user-generated-content.md)
>
> The write-side counterpart: stop LLM posts at the composer.

> [card] [Server verification](https://usefoil.com/docs/server-verification.md)
>
> Reference for both sealed-token and session-readback paths.

> [card] [Detection categories](https://usefoil.com/docs/detection-categories.md)
>
> How Foil combines its checks into a verdict.

> [card] [Going to production](https://usefoil.com/docs/going-to-production.md)
>
> Rollout plan for API-wide enforcement.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.