SkillHub

liang-tavily-crawl

v1.0.0

Crawl any website and save pages as local markdown files. Ideal for downloading documentation, knowledge bases, or web content for offline access or analysis.

Sourced from ClawHub, Authored by Liang

Installation

Please help me install the skill `liang-tavily-crawl` from SkillHub official store. npx skills add matthew77/liang-tavily-crawl

Tavily Crawl

Crawl websites to extract content from multiple pages. Ideal for documentation, knowledge bases, and site-wide content extraction.

Authentication

Get your API key at https://tavily.com and add to your OpenClaw config:

{
  "skills": {
    "entries": {
      "tavily-crawl": {
        "enabled": true,
        "apiKey": "tvly-YOUR_API_KEY_HERE"
      }
    }
  }
}

Or set in environment variable:

export TAVILY_API_KEY="tvly-YOUR_API_KEY_HERE"

Quick Start

Using the Script

node {baseDir}/scripts/crawl.mjs "https://docs.example.com"
node {baseDir}/scripts/crawl.mjs "https://docs.example.com" --output ./docs
node {baseDir}/scripts/crawl.mjs "https://example.com" --depth 2 --limit 50

Examples

# Basic crawl
node {baseDir}/scripts/crawl.mjs "https://docs.example.com"

# Deeper crawl with limits
node {baseDir}/scripts/crawl.mjs "https://docs.example.com" --depth 2 --limit 50

# Save to files
node {baseDir}/scripts/crawl.mjs "https://docs.example.com" --depth 2 --output ./docs

# Focused crawl with path filters
node {baseDir}/scripts/crawl.mjs "https://example.com" --depth 2 
  --select "/docs/.*" --exclude "/blog/.*"

# With semantic instructions
node {baseDir}/scripts/crawl.mjs "https://docs.example.com" 
  --instructions "Find API documentation" --chunks 3

Options

Option Description Default
--depth <n> Crawl depth (1-5) 1
--breadth <n> Links per page 20
--limit <n> Total pages cap 50
--output <dir> Save pages to directory -
--instructions <text> Natural language guidance -
--chunks <n> Chunks per page (1-5, requires instructions) -
--depth-mode <mode> Extract depth: basic or advanced basic
--select <pattern> Regex pattern to include -
--exclude <pattern> Regex pattern to exclude -
--timeout <sec> Max wait time (10-150 seconds) 150
--json Output raw JSON false

Depth vs Performance

Depth Typical Pages Time
1 10-50 Seconds
2 50-500 Minutes
3 500-5000 Many minutes

Start with --depth 1 and increase only if needed.

Crawl for Context vs Data Collection

For agentic use (feeding results into context): Always use --instructions + --chunks. This returns only relevant chunks instead of full pages, preventing context window explosion.

For data collection (saving to files): Omit --chunks to get full page content.

Tips

  • Always use --chunks for agentic workflows - prevents context explosion when feeding results to LLMs
  • Omit --chunks only for data collection - when saving full pages to files
  • Start conservative (--depth 1, --limit 20) and scale up
  • Use path patterns to focus on relevant sections
  • Always set a --limit to prevent runaway crawls