firecrawl-download — CLI для сбора и извлечения данных с сайтов

# 🔥 Firecrawl CLI Command-line interface for Firecrawl. Scrape, crawl, and extract data from any website directly from your terminal. ## Installation ```bash npm install -g firecrawl-cli ``` Or set up everything in one command (install CLI globally, authenticate, and add skills across all detected coding editors): ```bash npx -y firecrawl-cli@latest init -y --browser ``` - `-y` runs setup non-interactively - `--browser` opens the browser for Firecrawl authentication automatically - skills install globally to every detected AI coding agent by default ### Setup Skills and MCP If you are using an AI coding agent like Claude Code, you can also install the skill individually with: ```bash firecrawl setup skills ``` This installs skills globally across all detected coding editors by default. Use `--agent <agent>` to scope it to one editor. To install the Firecrawl MCP server into your editors (Cursor, Claude Code, VS Code, etc.): ```bash firecrawl setup mcp ``` Or directly via npx: ```bash npx skills add firecrawl/cli --full-depth --global --all npx add-mcp "npx -y firecrawl-mcp" --name firecrawl ``` ## Quick Start Just run a command - the CLI will prompt you to authenticate if needed: ```bash firecrawl https://example.com ``` ## Authentication On first run, you'll be prompted to authenticate: ``` 🔥 firecrawl cli Turn websites into LLM-ready data Welcome! To get started, authenticate with your Firecrawl account. 1. Login with browser (recommended) 2. Enter API key manually Tip: You can also set FIRECRAWL_API_KEY environment variable Enter choice [1/2]: ``` ### Authentication Methods ```bash # Interactive (prompts automatically when needed) firecrawl # Browser login firecrawl login # Direct API key firecrawl login --api-key fc-your-api-key # Environment variable export FIRECRAWL_API_KEY=fc-your-api-key # Per-command API key firecrawl scrape https://example.com --api-key fc-your-api-key ``` ### Self-hosted / Local Development For self-hosted Firecrawl instances or local development, use the `--api-url` option: ```bash # Use a local Firecrawl instance (no API key required) firecrawl --api-url http://localhost:3002 scrape https://example.com # Or set via environment variable export FIRECRAWL_API_URL=http://localhost:3002 firecrawl scrape https://example.com # Self-hosted with API key firecrawl --api-url https://firecrawl.mycompany.com --api-key fc-xxx scrape https://example.com ``` When using a custom API URL (anything other than `https://api.firecrawl.dev`), authentication is automatically skipped, allowing you to use local instances without an API key. --- ## Commands ### `scrape` - Scrape URLs Extract content from any webpage. Pass multiple URLs to scrape them concurrently -- each result is saved to `.firecrawl/` automatically. ```bash # Basic usage (outputs markdown) firecrawl https://example.com firecrawl scrape https://example.com # Get raw HTML firecrawl https://example.com --html firecrawl https://example.com -H # Multiple formats (outputs JSON) firecrawl https://example.com --format markdown,links,images # Save to file firecrawl https://example.com -o output.md firecrawl https://example.com --format json -o data.json --pretty # Multiple URLs (scraped concurrently, each saved to .firecrawl/) firecrawl scrape https://firecrawl.dev https://firecrawl.dev/blog https://docs.firecrawl.dev ``` #### Scrape Options | Option | Description | | -------------------------- | ------------------------------------------------------- | | `-f, --format <formats>` | Output format(s), comma-separated | | `-H, --html` | Shortcut for `--format html` | | `-S, --summary` | Shortcut for `--format summary` | | `--only-main-content` | Extract only main content (removes navs, footers, etc.) | | `--wait-for <ms>` | Wait time before scraping (for JS-rendered content) | | `--screenshot` | Take a screenshot | | `--full-page-screenshot` | Take a full page screenshot | | `--include-tags <tags>` | Only include specific HTML tags | | `--exclude-tags <tags>` | Exclude specific HTML tags | | `--max-age <milliseconds>` | Maximum age of cached content in milliseconds | | `-o, --output <path>` | Save output to file | | `--json` | Output as JSON format | | `--pretty` | Pretty print JSON output | | `--timing` | Show request timing info | #### Available Formats | Format | Description | | ---------------- | ---------------------------- | | `markdown` | Clean markdown (default) | | `html` | Cleaned HTML | | `rawHtml` | Original HTML | | `links` | All links on the page | | `images` | All images on the page | | `screenshot` | Screenshot as base64 | | `summary` | AI-generated summary | | `json` | Structured JSON extraction | | `changeTracking` | Track changes on the page | | `attributes` | Page attributes and metadata | | `branding` | Brand identity extraction | #### Examples ```bash # Extract only main content as markdown firecrawl https://blog.example.com --only-main-content # Wait for JS to render, then scrape firecrawl https://spa-app.com --wait-for 3000 # Get all links from a page firecrawl https://example.com --format links # Screenshot + markdown firecrawl https://example.com --format markdown --screenshot # Extract specific elements only firecrawl https://example.com --include-tags article,main # Exclude navigation and ads firecrawl https://example.com --exclude-tags nav,aside,.ad ``` --- ### `search` - Search the web Search the web and optionally scrape content from search results. ```bash # Basic search firecrawl search "firecrawl web scraping" # Limit results firecrawl search "AI news" --limit 10 # Search news sources firecrawl search "tech startups" --sources news # Search images firecrawl search "landscape photography" --sources images # Multiple sources firecrawl search "machine learning" --sources web,news,images # Filter by category (GitHub, research papers, PDFs) firecrawl search "web scraping python" --categories github firecrawl search "transformer architecture" --categories research firecrawl search "machine learning" --categories github,research # Time-based search firecrawl search "AI announcements" --tbs qdr:d # Past day firecrawl search "tech news" --tbs qdr:w # Past week # Location-based search firecrawl search "restaurants" --location "San Francisco,California,United States" firecrawl search "local news" --country DE # Search and scrape results firecrawl search "firecrawl tutorials" --scrape firecrawl search "API documentation" --scrape --scrape-formats markdown,links # Output as pretty JSON firecrawl search "web scraping" ``` #### Search Options | Option | Description | | ---------------------------- | ------------------------------------------------------------------------------------------- | | `--limit <n>` | Maximum results (default: 5, max: 100) | | `--sources <sources>` | Comma-separated: `web`, `images`, `news` (default: web) | | `--categories <categories>` | Comma-separated: `github`, `research`, `pdf` | | `--tbs <value>` | Time filter: `qdr:h` (hour), `qdr:d` (day), `qdr:w` (week), `qdr:m` (month), `qdr:y` (year) | | `--location <location>` | Geo-targeting (e.g., "Germany", "San Francisco,California,United States") | | `--country <code>` | ISO country code (default: US) | | `--timeout <ms>` | Timeout in milliseconds (default: 60000) | | `--ignore-invalid-urls` | Exclude URLs invalid for other Firecrawl endpoints | | `--scrape` | Enable scraping of search results | | `--scrape-formats <formats>` | Scrape formats when `--scrape` enabled (default: markdown) | | `--only-main-content` | Include only main content when scraping (default: true) | | `-o, --output <path>` | Save to file | | `--json` | Output as compact JSON | #### Examples ```bash # Research a topic with recent results firecrawl search "React Server Components" --tbs qdr:m --limit 10 # Find GitHub repositories firecrawl search "web scraping library" --categories github --limit 20 # Search and get full content firecrawl search "firecrawl documentation" --scrape --scrape-formats markdown --json -o results.json # Find research papers firecrawl search "large language models" --categories research --json # Search with location targeting firecrawl search "best coffee shops" --location "Berlin,Germany" --country DE # Get news from the past week firecrawl search "AI startups funding" --sources news --tbs qdr:w --limit 15 ``` --- ### `map` - Discover all URLs on a website Quickly discover all URLs on a website without scraping content. ```bash # List all URLs (one per line) firecrawl map https://example.com # Output as JSON firecrawl map https://example.com --json # Search for specific URLs firecrawl map https://example.com --search "blog" # Limit results firecrawl map https://example.com --limit 500 ``` #### Map Options | Option | Description | | --------------------------- | --------------------------------- | | `--limit <n>` | Maximum URLs to discover | | `--search <query>` | Filter URLs by search query | | `--sitemap <mode>` | `include`, `skip`, or `only` | | `--include-subdomains` | Include subdomains | | `--ignore-query-parameters` | Dedupe URLs with different params | | `--timeout <seconds>` | Request timeout | | `--json` | Output as JSON | | `-o, --output <path>` | Save to file | #### Examples ```bash # Find all product pages firecrawl map https://shop.example.com --search "product" # Get sitemap URLs only firecrawl map https://example.com --sitemap only # Save URL list to file firecrawl map https://example.com -o urls.txt # Include subdomains firecrawl map https://example.com --include-subdomains --limit 1000 ``` --- ### `crawl` - Crawl an entire website Crawl multiple pages from a website. ```bash # Start a crawl (returns job ID) firecrawl crawl https://example.com # Wait for crawl to complete firecrawl crawl https://example.com --wait # With progress indicator firecrawl crawl https://example.com --wait --progress # Check crawl status firecrawl crawl <job-id> # Limit pages firecrawl crawl https://example.com --limit 100 --max-depth 3 ``` #### Crawl Options | Option | Description | | --------------------------- | ---------------------------------------- | | `--wait` | Wait for crawl to complete | | `--progress` | Show progress while waiting | | `--limit <n>` | Maximum pages to crawl | | `--max-depth <n>` | Maximum crawl depth | | `--include-paths <paths>` | Only crawl matching paths | | `--exclude-paths <paths>` | Skip matching paths | | `--sitemap <mode>` | `include`, `skip`, or `only` | | `--allow-subdomains` | Include subdomains | | `--allow-external-links` | Follow external links | | `--crawl-entire-domain` | Crawl entire domain | | `--ignore-query-parameters` | Treat URLs with different params as same | | `--delay <ms>` | Delay between requests | | `--max-concurrency <n>` | Max concurrent requests | | `--timeout <seconds>` | Timeout when waiting | | `--poll-interval <seconds>` | Status check interval | #### Examples ```bash # Crawl blog section only firecrawl crawl https://example.com --include-paths /blog,/posts # Exclude admin pages firecrawl crawl https://example.com --exclude-paths /admin,/login # Crawl with rate limiting firecrawl crawl https://example.com --delay 1000 --max-concurrency 2 # Deep crawl with high limit firecrawl crawl https://example.com --limit 1000 --max-depth 10 --wait --progress # Save results firecrawl crawl https://example.com --wait -o crawl-results.json --pretty ``` --- ### `credit-usage` - Check your credits ```bash # Show credit usage firecrawl credit-usage # Output as JSON firecrawl credit-usage --json --pretty ``` --- ### `agent` - AI-powered web data extraction Run an AI agent that autonomously browses and extracts structured data from the web based on natural language prompts. > **Note:** Agent tasks typically take **2 to 5 minutes** to complete, and sometimes longer for complex extractions. Use sparingly and consider `--max-credits` to limit costs. ```bash # Basic usage (returns job ID immediately) firecrawl agent "Find the pricing plans for Firecrawl" # Wait for completion firecrawl agent "Extract all product names and prices from this store" --wait # Focus on specific URLs firecrawl agent "Get the main features listed" --urls https://example.com/features # Use structured output with JSON schema firecrawl agent "Extract company info" --schema '{"type":"object","properties":{"name":{"type":"string"},"employees":{"type":"number"}}}' # Load schema from file firecrawl agent "Extract product data" --schema-file ./product-schema.json --wait # Check status of an existing job firecrawl agent <job-id> firecrawl agent <job-id> --wait ``` #### Agent Options | Option | Description | | --------------------------- | ------------------------------------------------------------- | | `--urls <urls>` | Comma-separated URLs to focus extraction on | | `--model <model>` | `spark-1-mini` (default, cheaper) or `spark-1-pro` (accurate) | | `--schema <json>` | JSON schema for structured output (inline JSON string) | | `--schema-file <path>` | Path to JSON schema file for structured output | | `--max-credits <number>` | Maximum credits to spend (job fails if exceeded) | | `--status` | Check status of existing agent job | | `--wait` | Wait for agent to complete before returning results | | `--poll-interval <seconds>` | Polling interval in seconds when waiting (default: 5) | | `--timeout <seconds>` | Timeout in seconds when waiting (default: no timeout) | | `-o, --output <path>` | Save output to file | | `--json` | Output as JSON format | | `--pretty` | Pretty print JSON output | #### Examples ```bash # Research task with timeout firecrawl agent "Find the top 5 competitors of Notion and their pricing" --wait --timeout 300 # Extract data with cost limit firecrawl agent "Get all blog post titles and dates" --urls https://blog.example.com --max-credits 100 --wait # Use higher accuracy model for complex extraction firecrawl agent "Extract detailed technical specifications" --model spark-1-pro --wait --pretty # Save structured results to file firecrawl agent "Extract contact information" --schema-file ./contact-schema.json --wait -o contacts.json --pretty # Check job status without waiting firecrawl agent abc123-def456-... --json # Poll a running job until completion firecrawl agent abc123-def456-... --wait --poll-interval 10 ``` --- ### `browser` - Browser sandbox sessions (Deprecated) > **Deprecated:** Prefer `scrape` + `interact` instead. Interact lets you scrape a page and then click, fill forms, and navigate without managing sessions manually. See the `interact` command. Launch and control cloud browser sessions. By default, commands are sent to agent-browser (pre-installed in every sandbox). Use `--python` or `--node` to run Playwright code directly instead. ```bash # 1. Launch a session firecrawl browser launch --stream # 2. Execute agent-browser commands (default) firecrawl browser execute "open https://example.com" firecrawl browser execute "snapshot" firecrawl browser execute "click @e5" firecrawl browser execute "scrape" # 3. Execute Playwright Python or JavaScript firecrawl browser execute --python "await page.goto('https://example.com'); print(await page.title())" firecrawl browser execute --node "await page.goto('https://example.com'); await page.title()" # 4. List sessions firecrawl browser list # 5. Close firecrawl browser close ``` #### Launch Options | Option | Description | | ---------------------------- | ------------------------------------------- | | `--ttl <seconds>` | Total session TTL in seconds (default: 300) | | `--ttl-inactivity <seconds>` | Inactivity TTL in seconds | | `--stream` | Enable live view streaming | | `-o, --output <path>` | Save output to file | | `--json` | Output as JSON format | #### Execute Options | Option | Description | | --------------------- | ------------------------------------------------------------------ | | `--python` | Execute as Playwright Python code | | `--node` | Execute as Playwright JavaScript code | | `--bash` | Execute bash commands in the sandbox (agent-browser pre-installed) | | `--session <id>` | Target a specific session (auto-saved on launch) | | `-o, --output <path>` | Save output to file | | `--json` | Output as JSON format | By default (no flag), commands are sent to agent-browser. `--python`, `--node`, and `--bash` are mutually exclusive. #### Examples ```bash # agent-browser commands (default mode) firecrawl browser execute "open https://example.com" firecrawl browser execute "snapshot" firecrawl browser execute "click @e5" firecrawl browser execute "fill @e3 'search query'" firecrawl browser execute "scrape" # Playwright Python firecrawl browser execute --python "await page.goto('https://example.com'); print(await page.title())" # Playwright JavaScript firecrawl browser execute --node "await page.goto('https://example.com'); await page.title()" # Bash (arbitrary commands in the sandbox) firecrawl browser execute --bash "ls /tmp" # Launch with extended TTL firecrawl browser launch --ttl 900 --ttl-inactivity 120 # JSON output firecrawl browser execute --json "snapshot" ``` --- ### `config` - Configure settings ```bash # Configure with custom API URL firecrawl config --api-url https://firecrawl.mycompany.com firecrawl config --api-url http://localhost:3002 --api-key fc-xxx ``` ### `view-config` - View current configuration ```bash # View current configuration and authentication status firecrawl view-config ``` Shows authentication status and stored credentials location. --- ### `login` / `logout` ```bash # Login firecrawl login firecrawl login --method browser firecrawl login --method manual firecrawl login --api-key fc-xxx # Login to self-hosted instance firecrawl login --api-url https://firecrawl.mycompany.com firecrawl login --api-url http://localhost:3002 --api-key fc-xxx # Logout firecrawl logout ``` --- ## Global Options These options work with any command: | Option | Description | | --------------------- | ------------------------------------------------------ | | `--status` | Show version, auth, concurrency, and credits | | `-k, --api-key <key>` | Use specific API key | | `--api-url <url>` | Use custom API URL (for self-hosted/local development) | | `-V, --version` | Show version | | `-h, --help` | Show help | ### Check Status ```bash firecrawl --status ``` ``` 🔥 firecrawl cli v1.4.0 ● Authenticated via stored credentials Concurrency: 0/100 jobs (parallel scrape limit) Credits: 500,000 / 1,000,000 (50% left this cycle) ``` --- ## Output Handling ### Stdout vs File ```bash # Output to stdout (default) firecrawl https://example.com # Pipe to another command firecrawl https://example.com | head -50 # Save to file firecrawl https://example.com -o output.md # JSON output firecrawl https://example.com --format links --pretty ``` ### Format Behavior - **Single format**: Outputs raw content (markdown text, HTML, etc.) - **Multiple formats**: Outputs JSON with all requested data ```bash # Raw markdown output firecrawl https://example.com --format markdown # JSON output with multiple formats firecrawl https://example.com --format markdown,links,images ``` --- ## Tips & Tricks ### Scrape multiple URLs ```bash # Just pass multiple URLs -- results are saved to .firecrawl/ firecrawl scrape https://firecrawl.dev https://firecrawl.dev/blog https://docs.firecrawl.dev ``` ### Combine with other tools ```bash # Extract links and process with jq firecrawl https://example.com --format links | jq '.links[].url' # Convert to PDF (with pandoc) firecrawl https://example.com | pandoc -o document.pdf # Search within scraped content firecrawl https://example.com | grep -i "keyword" ``` ### CI/CD Usage ```bash # Set API key via environment export FIRECRAWL_API_KEY=${{ secrets.FIRECRAWL_API_KEY }} firecrawl crawl https://docs.example.com --wait -o docs.json # Use self-hosted instance export FIRECRAWL_API_URL=${{ secrets.FIRECRAWL_API_URL }} firecrawl scrape https://example.com -o output.md ``` --- ## `download` - Bulk Site Download A convenience command that combines `map` + `scrape` to save a site as local files. Maps the site first to discover pages, then scrapes each one into nested directories under `.firecrawl/`. All [scrape options](#scrape-options) work with download. Run without flags for an interactive wizard that walks you through format, screenshot, and path selection. ```bash # Interactive wizard (picks format, screenshots, paths for you) firecrawl download https://docs.firecrawl.dev # Download with screenshots firecrawl download https://docs.firecrawl.dev --screenshot --limit 20 -y # Full page screenshots firecrawl download https://docs.firecrawl.dev --full-page-screenshot --limit 20 -y # Multiple formats (each saved as its own file per page) firecrawl download https://docs.firecrawl.dev --format markdown,links --screenshot --limit 20 -y # Creates per page: index.md + links.txt + screenshot.png # Download as HTML firecrawl download https://docs.firecrawl.dev --html --limit 20 -y # Main content only firecrawl download https://docs.firecrawl.dev --only-main-content --limit 50 -y # Filter to specific paths firecrawl download https://docs.firecrawl.dev --include-paths "/features,/sdks" # Skip localized pages firecrawl download https://docs.firecrawl.dev --exclude-paths "/zh,/ja,/fr,/es,/pt-BR" # Include subdomains firecrawl download https://firecrawl.dev --allow-subdomains # Combine everything firecrawl download https://docs.firecrawl.dev --include-paths "/features,/sdks" --exclude-paths "/zh,/ja,/fr,/es,/pt-BR" --only-main-content --screenshot -y ``` #### Download Options | Option | Description | | ------------------------- | ---------------------------------------------- | | `--limit <number>` | Max pages to download | | `--search <query>` | Filter pages by search query | | `--include-paths <paths>` | Only download matching paths (comma-separated) | | `--exclude-paths <paths>` | Skip matching paths (comma-separated) | | `--allow-subdomains` | Include subdomains when mapping | | `-y, --yes` | Skip confirmation prompt and wizard | All [scrape options](#scrape-options) also work with download (formats, screenshots, tags, geo-targeting, etc.) #### Output Structure Each format is saved as its own file per page: ``` .firecrawl/ docs.firecrawl.dev/ features/ scrape/ index.md # markdown content links.txt # one link per line screenshot.png # actual PNG image crawl/ index.md screenshot.png sdks/ python/ index.md ``` --- ## Telemetry The CLI collects anonymous usage data during authentication to help improve the product: - CLI version, OS, and Node.js version - Detect development tools (e.g., Cursor, VS Code, Claude Code) **No command data, URLs, or file contents are collected via the CLI.** To disable telemetry, set the environment variable: ```bash export FIRECRAWL_NO_TELEMETRY=1 ``` --- ## Experimental: AI Workflows Launch pre-built AI workflows that combine Firecrawl's web capabilities with your coding agent. One command spins up an interactive session with the right system prompt, tools, and instructions -- like `ollama run` but for web research agents. All workflows spawn parallel subagents to divide the work and finish faster. ```bash # Claude Code (available now) firecrawl claude competitor-analysis firecrawl claude deep-research firecrawl claude lead-research firecrawl claude seo-audit firecrawl claude qa # Codex and OpenCode -- coming soon firecrawl codex competitor-analysis firecrawl opencode competitor-analysis ``` See the full documentation: **[Experimental Workflows ->](src/commands/experimental/README.md)** --- ## Testing Workflows Locally After building the CLI (`pnpm run build`), every workflow works with all three backends — just swap the command name: ```bash # Help firecrawl claude --help firecrawl codex --help firecrawl opencode --help ``` ### QA Testing ```bash firecrawl claude qa https://myapp.com firecrawl codex qa https://myapp.com firecrawl opencode qa https://myapp.com ``` ### Product Demo Walkthrough ```bash firecrawl claude demo https://resend.com firecrawl codex demo https://neon.tech firecrawl opencode demo https://linear.app ``` ### Competitor Analysis ```bash firecrawl claude competitor-analysis https://firecrawl.dev firecrawl codex competitor-analysis https://crawlee.dev firecrawl opencode competitor-analysis https://apify.com ``` ### Deep Research ```bash firecrawl claude deep-research "RAG pipeline data ingestion tools" firecrawl codex deep-research "web scraping best practices 2025" firecrawl opencode deep-research "browser automation frameworks comparison" ``` ### Other Workflows ```bash # Lead research firecrawl claude lead-research "Vercel" firecrawl codex lead-research "Stripe" # SEO audit firecrawl opencode seo-audit https://example.com # Knowledge base firecrawl claude knowledge-base https://docs.langchain.com # Research papers firecrawl codex research-papers "web scraping compliance HIPAA" # Shopping firecrawl claude shop "best mechanical keyboard for developers" ``` ### Natural Language (no workflow name) ```bash firecrawl claude "scrape the firecrawl docs and summarize" firecrawl codex "find pricing for crawlee vs scrapy" firecrawl opencode "compare Firecrawl and Apify features" ``` Add `-y` to any command to auto-approve tool permissions (maps to `--dangerously-skip-permissions` for Claude, `--full-auto` for Codex). ### Live View Use `firecrawl scrape <url>` + `firecrawl interact` to interact with pages. For advanced use cases requiring a raw CDP session, you can still use `firecrawl browser launch --json` to get a live view URL: ```bash # Preferred: scrape + interact workflow firecrawl scrape https://myapp.com firecrawl interact --prompt "Click on the login button and fill in the form" # Advanced: Launch a browser session and grab the live view URL LIVE_URL=$(firecrawl browser launch --json | jq -r '.liveViewUrl') # Pass it to Claude Code claude --append-system-prompt "A cloud browser session is running. Live view: $LIVE_URL -- use `firecrawl interact` to interact with scraped pages." --dangerously-skip-permissions "QA test https://myapp.com" # Or use the built-in workflow commands firecrawl claude demo https://resend.com ``` ### Prerequisites Each backend requires its CLI to be installed separately: | Backend | Install | | -------- | ----------------------------------------------------- | | Claude | `npm install -g @anthropic-ai/claude-code` | | Codex | `npm install -g @openai/codex` | | OpenCode | [opencode.ai/docs/cli](https://opencode.ai/docs/cli/) | --- ## Documentation For more details, visit the [Firecrawl Documentation](https://docs.firecrawl.dev).