cto-legends · module specification · intelligence · v0.3.2
legends-firecrawl: Official Firecrawl Web Extraction, Alexandria Catalog & Evidence Bank
Agent-first CLI wrapper (lfc and lax) for official Firecrawl web extraction and Alexandria queries with full local JSON evidence banking.
GitHub Repository (https://github.com/avalonreset/legends-firecrawl) · Handoff: cto-legends handoff legends-firecrawl
legends-firecrawl provides two deterministic CLI entry points—lfc for official Firecrawl web operations (scrape, map, search, crawl-preview, crawl, crawl-status, credits) and lax for Alexandria structured data queries (search, inspect, audit, query, captures)—paired with an offline-verifiable research evidence bank. Rather than streaming truncated web scrapes through an ambient MCP server and discarding the raw payload after a single turn, legends-firecrawl follows a 'collect once, inspect offline' architecture: every live scrape, map, search, crawl, and confirmed Alexandria query saves the complete returned JSON to var/captures/firecrawl/ alongside a relative Markdown reference card in vault/captures/firecrawl/.
In release v0.3.2, execution is strictly scoped to official Firecrawl services with all experimental native/direct-source bypasses retired. Alexandria queries default to a zero-charge request preview (--preview) and require explicit --confirm before live submission; multi-page web crawls similarly provide an offline lfc crawl-preview gate and require explicit --confirm with bounded --limit and --max-depth controls. Default captures live outside the module checkout under ~/.legends-firecrawl/research (or LEGENDS_FIRECRAWL_CAPTURE_ROOT), where lfc evidence commands export, hash-verify, inventory, slice, and evaluate freshness for reuse or legends-empire vault handoff without re-querying the provider.
Deterministic Execution Boundary
Why Raw Frontier LLM Prompts Cannot Replace Deterministic Execution
A raw LLM chat session or basic fetch tool cannot render JavaScript-heavy public web pages into clean Markdown with link inventories, bound multi-page crawl depth before execution, browse 114 Alexandria provider contracts offline, or retain immutable JSON provenance across sessions. When agents rely on ephemeral web-browsing tools without a capture-first CLI layer, they repeatedly scrape the same URLs, lose retrieval timestamps, and cannot prove whether an empty or partial result was a site error or a pagination limit.
- Capture-First Provenance Outside Replaceable Installs: Every web and Alexandria operation writes the complete untouched JSON response and `_capture` metadata paths to `~/.legends-firecrawl/research` (or `LEGENDS_FIRECRAWL_CAPTURE_ROOT`) by default, ensuring module upgrades never overwrite client research and agents can re-read any field offline.
- Preview-First Alexandria & Bounded Crawl Confirmation Gates: Running `lax query <provider> <endpoint>` or `lfc crawl-preview <url>` inspects parameters and scope without network submission. Live execution of Alexandria queries and multi-page crawls is blocked unless `--confirm` is explicitly supplied.
- Offline Catalog Discovery Across 114 Providers & 797 Capabilities: `lax search`, `lax inspect <provider> [--full]`, and `lax audit` query the bundled local JSON catalog snapshot offline, returning exact cached schemas and file paths without truncating the underlying archive or requiring API credentials.
- Portable SHA-256 Evidence Bank & Scoped Reuse Checks: `lfc evidence export-capture` creates `legends-firecrawl-evidence/v1` packages binding `response.json`, `manifest.json`, and `README.md` hashes, workspace identity, request settings, and `observed_at` timestamps, while `lfc evidence reuse` enforces exact request matching and `--max-age-hours` ceilings.
CLI Subcommands & Capability Surface
lfc scrape <url> [--formats markdown,links] | lfc map <url> --limit 100 [--search <terms>] | lfc search <query> --limit 5 — Intent-Level Web Scrape, Map & Corroboration Search: Executes official Firecrawl web extraction via firecrawl-cli 1.23.3 and Python client workflows, automatically saving complete JSON captures and Markdown vault cards. Supports the Legends Packet formula: map a prospect domain with a 100-URL ceiling first, select high-signal pages, and scrape only those targets.lfc crawl-preview <url> --limit 25 --max-depth 2 [--include-paths /services] & lfc crawl <url> ... --confirm — Preview-Gated Bounded Multi-Page Crawling: Validates target domain, page limits, crawl depth, and path filters locally with crawl-preview before making any HTTP request. Live crawl submission requires --confirm and preserves both initial job submission and subsequent crawl-status responses in the capture store.lax search <term> | lax inspect <provider> [--full] | lax query <provider> <capability> [--confirm] — Offline Alexandria Catalog Inspection & Confirmed Queries: Searches and inspects the local snapshot of 114 Alexandria providers and 797 capabilities offline. Running lax query defaults to an offline request preview; adding --confirm dispatches the query through official Firecrawl Alexandria execution and records the full JSON capture and vault reference.lfc evidence export | export-capture | inventory | find | verify | view <pkg> [--select data.markdown | --full] — Portable Evidence Export, Verification & Selective Views: Packages standalone responses or raw captures into content-addressed `legends-firecrawl-evidence/v1` directories. `inventory` scans bounded manifests without reading full response blobs (`response_integrity: not_checked`), while `view` verifies all three package files before projecting dotted fields like `data.markdown`.lfc evidence reuse <pkg> --workspace <id> --endpoint <op> --request-file <req.json> --max-age-hours 24 — Deterministic Freshness & Reuse Eligibility Gate: Checks offline whether an existing verified package matches the exact client workspace, operation, and request parameters within a finite freshness window and completed response state. Exits 0 when eligible for semantic review and 2 when stale, incomplete, or mismatched, never auto-purchasing replacement data.pwsh -File bin/doctor.ps1 & lfc credits — Environment Readiness & Blocked Vendor Command Guardrails: Verifies Python runtime, Node.js availability, firecrawl-cli@1.23.3 installation, and FIRECRAWL_API_KEY configuration via bin/doctor.ps1, while deterministically blocking interactive vendor launcher/init commands and ambient MCP fallbacks.
Raw Frontier LLM vs. legends-firecrawl Execution Matrix
- Web Extraction Persistence: Raw LLM (Streams scraped Markdown into transient chat context; re-scrapes the same URL when needed in a later turn.) vs. Module (Saves full JSON to var/captures/firecrawl/ and Markdown cards to vault/captures/firecrawl/ under ~/.legends-firecrawl/research.) — CLI output embeds `_capture` paths (or `--receipt` paths); `lfc evidence verify` checks SHA-256 integrity across package files.
- Crawl & Alexandria Safety: Raw LLM (Launches unbounded recursive site crawls or paid Alexandria queries without previewing depth or parameters.) vs. Module (Enforces `lfc crawl-preview` and default `--preview` on `lax query`; requires explicit `--confirm` for crawl and Alexandria execution.) — Behavioral evals (`scripts/run_behavioral_evals.py`) and contract tests verify zero network calls without `--confirm`.
- Execution Provenance & Scope: Raw LLM (Mixes unofficial scrapers, unverified third-party APIs, and silent fallbacks when a page or endpoint fails.) vs. Module (Executes official Firecrawl routes only (all native/direct-source bypasses retired in v0.3.2) and records exact response states.) — `legends-firecrawl-evidence/v1` manifests bind request settings, `observed_at` UTC timestamps, and response state.
- Context Window Management: Raw LLM (Loads entire raw crawl arrays or 114-provider catalog dumps into prompt tokens.) vs. Module (Uses compact catalog discovery (`lax search`) and verified field projection (`lfc evidence view --select data.markdown`).) — Bounded views report exact omission counts while leaving underlying `response.json` bytes untouched.
Explicit Safety Boundaries & Exclusions
Operator FAQ
- Why were native direct-source bypasses removed in legends-firecrawl v0.3.2?
- Historical October 2026 audits showed that direct native substitutions across third-party APIs suffered from schema drift, partial field matches, and rate-limit failures. Release v0.3.2 enforces 100% official Firecrawl execution so every command has predictable contracts, transparent provider billing, and uniform capture provenance.
- How do lfc and lax prevent accidental credit consumption during agent exploration?
- Catalog discovery (`lax search`, `lax inspect`, `lax audit`) and `lfc crawl-preview` run completely offline. Alexandria queries (`lax query`) default to `--preview` mode and refuse to submit live requests without `--confirm`, while `lfc evidence reuse` checks whether fresh local captures already satisfy the request before any new call is made.
- Where are Firecrawl captures stored and how do they hand off to legends-empire?
- By default, captures are stored outside the module installation at `~/.legends-firecrawl/research` (or `LEGENDS_FIRECRAWL_CAPTURE_ROOT`). `lfc evidence export-capture` packages the raw JSON, manifest, and README with SHA-256 hashes so `legends-empire` can stage and ingest verified research packages without duplicating raw scrape dumps inside the knowledge vault.
All 19 CTO Legends Packages (1 Router + 18 Modules)
cto-legends (v0.2.5) — Single-skill Python CLI router and isolated environment manager that resolves 18 SHA-256-pinned modules on demand without MCP startup bloat.legends-empire (v0.3.2) — SHA-256 Markdown knowledge foundation with recoverable NTFS and POSIX transactions, .raw/ evidence quarantine, and read-only vault stewardship.legends-hetzner (v0.1.3) — Hetzner Cloud CLI toolkit with Cost-Optimized CX/CAX stock radar, Fractional CTO sizing, SHA-256 server plans, 10-check preflight, and VPS sniping.legends-coolify (v0.2.1) — Read-only Coolify PaaS CLI for self-hosted app inventory, secret-free env key inspection, backup schedule audits, deployment history, and runbooks.legends-cloudflare (v0.1.1) — Zero-dependency Cloudflare DNS CLI enforcing SHA-256 preview-first single-record upserts, exact-ID deletes, setting preservation, and API readback.legends-astro (v0.2.1) — Astro v7 migration CLI that extracts HTML and WordPress source contracts, scaffolds Content Layer collections, and verifies dist/ build parity.legends-gohighlevel (v0.3.0) — Python 3.11+ GoHighLevel CLI for API v2 CRM, Workflows v3, Agent Studio AI, and 5-layer agency RBAC/1.05x rebilling with SHA-256 preview and readback.legends-geogrid (v0.2.2) — Circular-lattice Google Maps rank tracker, GBP name-stuffing audit workbench, and 18-page vector PDF strategy studio for coding agents.legends-dataforseo (v0.1.6) — Zero-dependency Python API and CLI for DataForSEO v3 SERP, Maps, GBP, and keyword research with offline previews and evidence banking.legends-firecrawl (v0.3.2) — Agent-first CLI wrapper (lfc and lax) for official Firecrawl web extraction and Alexandria queries with full local JSON evidence banking.legends-github (v0.2.1) — Deterministic Python CLI for GitHub repository audits, DataForSEO keyword and SERP demand, 20-topic synthesis, non-destructive READMEs, and OG checks.legends-grant (v0.3.0) — Standard-library Python CLI for 56-jurisdiction US grant discovery, SQLite evidence banking, qualification checks, and proposal verification.legends-jev (v0.2.1) — Zero-dependency Python CLI for TypeSafe Jev 1.14 optical recognition (OCR), storefront signage checks, pixel bounding boxes, and typed UI decisions.legends-obs (v0.1.2) — Guarded Windows OBS Studio CLI over WebSocket v5 for hardware encoder planning, pre-mutation rollback snapshots, and FFprobe-verified recording canaries.legends-captions (v0.1.1) — Deterministic video caption QA and ASS rendering CLI with contextual ASR repair, NeMo Forced Aligner word timing, spoken acronym locks, and proof frames.legends-yt-dlp (v0.1.2) — Paced yt-dlp orchestration CLI with SHA-256 binary verification, channel item ledgers, opt-in Mullvad Lockdown guards, and Parakeet/NFA word-level search.legends-stable-audio-3 (v0.1.3) — Hardware-aware Stable Audio 3 CLI for VRAM-calibrated local Medium batches, guarded Large REST jobs, cue-trimmed streaming mixes, and float WAV mastering.legends-hyperyap (v0.1.1) — Offline desktop dictation runtime running INT8 ONNX NVIDIA Parakeet TDT 0.6B v2 locally with CUDA-first inference, active-window paste, and terminal hooks.legends-ambient-intelligence (v0.1.1) — Passive audio pipeline CLI for SHA-256 deduplicated USB recorder ingest, offline Whisper/NeMo Parakeet ASR, rollover recovery, and vault distillation.