flowcrawl
v1.0.2Stealth web scraper. Give it any URL and it punches through Cloudflare, bot detection, and WAFs automatically using a 3-tier cascade (plain HTTP → TLS spoof → full JS). No API keys, no proxies, no CDP Chrome. Free from the Flow team. Use when scraping any website, bypassing bot protection, spidering...
Sourced from ClawHub,
Authored by windseeker1111
Installation
Please help me install the skill `flowcrawl` from SkillHub official store.
npx skills add windseeker1111/flowcrawl
FlowCrawl
Scrape any website. Bypass any bot protection. Free.
Install Scrapling First
pip install scrapling
Scrapling installs Playwright automatically on first run. That's the only dependency.
Quick Usage
# Single URL — prints clean markdown to stdout
python3 ~/clawd/skills/flowcrawl/scripts/flowcrawl.py https://example.com
# Spider the whole site
python3 ~/clawd/skills/flowcrawl/scripts/flowcrawl.py https://example.com --deep
# Deep crawl with limits, save and combine
python3 ~/clawd/skills/flowcrawl/scripts/flowcrawl.py https://example.com --deep --limit 30 --combine
# JSON output — pipe into anything
python3 ~/clawd/skills/flowcrawl/scripts/flowcrawl.py https://example.com --json
Add Alias (Recommended)
echo 'alias flowcrawl="python3 ~/clawd/skills/flowcrawl/scripts/flowcrawl.py"' >> ~/.zshrc
source ~/.zshrc
Then just: flowcrawl https://example.com
How It Works
FlowCrawl uses a 3-tier fetcher cascade. Starts fast, escalates only when blocked:
| Tier | Method | Handles |
|---|---|---|
| 1 | Plain HTTP | Most sites, instant |
| 2 | Stealth + TLS spoof | Cloudflare, Imperva, basic WAFs |
| 3 | Full JS execution | SPAs, heavy JS, aggressive bot detection |
Auto-detects blocking (403, 503, "Just a moment...") and escalates silently.
All Options
| Flag | Description | Default |
|---|---|---|
--deep |
Spider whole site following internal links | off |
--depth N |
Max hop depth from start URL | 3 |
--limit N |
Max pages to crawl | 50 |
--combine |
Merge all pages into one file | off |
--format md|txt |
Output format | md |
--output DIR |
Output directory | ./flowcrawl-output |
--json |
Structured JSON output | off |
--quiet |
Suppress progress logs | off |