Threat Protection

​

Modes

• Off (default) — no checks are performed.
• Normal — URLs are checked against Google Web Risk, which flags pages and sites associated with malware, social engineering (phishing), and unwanted software. +2 credits per URL scanned.
• Zscaler — URLs are checked against your organization’s own Zscaler Internet Access (ZIA) tenant: the Zscaler-defined URL categories you choose to block, plus your custom URL categories and custom URL lists. See Zscaler mode below. No scan fees — classification runs against your own tenant.

​

Policy controls

• Custom blacklist — exact domains or globs (e.g. *.example.com) that are always blocked, without a classifier call.
• Custom whitelist — exact domains or globs that are always allowed. The whitelist wins over every other rule, so a domain you trust is never blocked.
• Blocked TLDs — top-level domains to block outright (e.g. zip), matched on label boundaries.
• Risk score threshold — the normalized score (0–100) at or above which a classifier verdict is treated as a block. Lower is stricter. The default is 75. Applies to Normal mode; Zscaler mode blocks by category instead.
• Failure policy — what to do when the classifier can’t be reached: block (closed, the default and recommended for a security control) or allow (open).

​

Zscaler mode

• Inline classification — URLs are classified through your tenant’s URL Lookup API into Zscaler-defined categories, and URLs in a category you have denied are blocked.
• Synced custom rules — your custom URL categories (URL lists and keywords) and your additions to Zscaler-defined categories are synced from the tenant on a schedule and evaluated by Firecrawl directly, because the ZIA lookup API does not return custom classifications. Entries removed in ZIA disappear on the next sync; a manual Sync now is available in the dashboard.

​

Connecting your tenant

​

Capacity and behavior

• Map results are evaluated against local rules only (your lists plus synced custom rules) rather than classified inline — one map can return thousands of URLs, and classifying them would burn the hourly budget on links that may never be fetched. Every URL still gets the full check when a scrape of it starts.
• Search results are classified inline and blocked results are removed, same as Normal mode; each unique result draws on the hourly lookup budget.

​

Configuring the policy

1. Open Enterprise Controls → Threat Protection.
2. Choose a mode, set your risk score threshold, and add any blacklist, whitelist, or blocked-TLD entries.
3. For Zscaler mode: enter the tenant connection, run the connection test, pick the categories to block, and set the sync interval.
4. Choose whether to allow per-request overrides, and set the failure policy.
5. Save. Changes take effect immediately — the next request is evaluated against the new policy.

​

Per-request overrides

{
  "url": "https://example.com",
  "threatProtection": {
    "mode": "normal",
    "riskScoreThreshold": 50,
    "blacklist": ["*.risky.example"]
  }
}

​

When a URL is blocked

{
  "success": false,
  "code": "unsafe_domain_blocked",
  "error": "This URL (https://risky.example/landing) is blocked by your organization's threat protection policy (rule: blacklist). If you believe this is a mistake, contact your organization administrator to adjust the policy (e.g. whitelist the domain)."
}

• Scrape, batch scrape, extract, agent — a blocked target returns the unsafe_domain_blocked error for that URL.
• Crawl — a blocked seed URL fails the request; blocked links discovered mid-crawl are skipped and the crawl continues.
• Search, map — blocked URLs are removed from the returned results rather than surfaced and refused.

​

Billing

• Decisions made entirely from your own policy (blacklist, whitelist, or blocked-TLD matches) do not call the classifier and are not charged a scan fee.
• A request that is blocked is still charged for the scan that produced the verdict.
• Scans are deduplicated within a single scrape: a redirect re-check that resolves to the same URL shares the original scan, while a redirect that lands on a different URL is a second scan.
• Crawls and batch scrapes check every page independently. Verdicts are never reused across pages — nothing about your traffic is stored (see above) — so in Normal mode expect +2 credits per scraped page. A link that is discovered mid-crawl and blocked bills its scan once per crawl, no matter how many pages link to it.
• Search and map scan each unique URL in the result set once per request, so their scan fees scale with the number of results scanned — which can slightly exceed the number returned when results are trimmed to your limit.

​

Error reference

​

Notes

• The policy is organization-wide: it applies to every API key and every endpoint automatically.
• The whitelist always wins, so a URL on an explicitly trusted domain is never blocked by the classifier or a TLD rule.
• The error code unsafe_domain_blocked is kept stable for compatibility even though checks are URL-level.
• With the failure policy set to closed (the default), a classifier outage causes affected requests to be blocked rather than silently allowed.
• With SIEM Audit Logging configured, every decision is visible in your audit trail: events carry the deciding rule, the classifier consulted, the threat categories, and — for Zscaler-classified URLs — a security_alert flag when the URL carried a security-alert classification.