Subchapter 2.1

Crawl

references/crawl.mdMarkdown10 KBView on GitHub

Crawl & Map API Reference

Feature	Crawl	Map
Returns	Full content	URLs only
Speed	Slower	Faster
Best for	RAG, deep analysis, documentation	Site structure discovery, URL collection

Crawl · Tavily SDK · tavily-ai/skills · Skills Docs

Parameter	Type	Default	Description
`url`	string	Required	Root URL to begin
`max_depth`	integer	1	Levels deep to crawl (1-5). Start with 1-2
`max_breadth`	integer	20	Links per page. 50-100 for focused crawls
`limit`	integer	50	Total pages cap
`instructions`	string	null	Natural language guidance (2 credits/10 pages)
`chunks_per_source`	integer	3	Chunks per page (1-5). Only with `instructions`
`extract_depth`	enum	`"basic"`	`"basic"` (1 credit/5 URLs) or `"advanced"` (2 credits/5 URLs)
`format`	enum	`"markdown"`	`"markdown"` or `"text"`
`select_paths`	array	null	Regex patterns to include
`exclude_paths`	array	null	Regex patterns to exclude
`select_domains`	array	null	Regex for domains to include
`exclude_domains`	array	null	Regex for domains to exclude
`allow_external`	boolean	true (crawl) / false (map)	Include external domain links
`include_images`	boolean	false	Include images (crawl only)
`include_favicon`	boolean	false	Include favicon URL (crawl only)
`include_usage`	boolean	false	Include credit usage info
`timeout`	float	150	Max wait (10-150 seconds)

response = client.crawl(
    url="https://docs.example.com",
    max_depth=2,
    instructions="Find all documentation about authentication and security",
    chunks_per_source=3  # Only top 3 relevant chunks per page
)

# Target specific sections
response = client.crawl(
    url="https://example.com",
    select_paths=["/docs/.*", "/api/.*", "/guides/.*"],
    exclude_paths=["/blog/.*", "/changelog/.*", "/private/.*"]
)

# Paginated content
response = client.crawl(
    url="https://example.com/blog",
    max_depth=2,
    select_paths=["/blog/.*", "/blog/page/.*"],
    exclude_paths=["/blog/tag/.*"]
)

# Stay within subdomain
response = client.crawl(
    url="https://docs.example.com",
    select_domains=["^docs.example.com$"],
    max_depth=2
)

# Exclude specific domains
response = client.crawl(
    url="https://example.com",
    exclude_domains=["^ads.example.com$", "^tracking.example.com$"]
)

response = client.crawl(
    url="https://example.com",
    max_depth=3,
    max_breadth=50,
    limit=200,
    select_paths=["/blog/.*", "/changelog/.*"],
    exclude_paths=["/private/.*", "/admin/.*"]
)

response = client.crawl(
    url="https://docs.example.com",
    max_depth=2,
    extract_depth="advanced",
    include_images=True,
    instructions="Extract all technical documentation and code examples"
)

response = client.crawl(
    url="https://example.com",
    max_depth=3,
    max_breadth=100,
    limit=1000,
    extract_depth="advanced",
    instructions="Find all mentions of GDPR and data protection policies"
)

# Step 1: Map to discover structure
map_result = client.map(
    url="https://docs.example.com",
    max_depth=2,
    instructions="Find all API docs and guides"
)

# Step 2: Filter discovered URLs
api_docs = [url for url in map_result["results"] if "/api/" in url]
guides = [url for url in map_result["results"] if "/guides/" in url]
print(f"Found {len(api_docs)} API docs, {len(guides)} guides")

# Step 3: Extract from filtered URLs
target_urls = api_docs + guides
response = client.extract(
    urls=target_urls[:20],  # Max 20 per extract call
    extract_depth="advanced",
    query="API endpoints and usage examples",
    chunks_per_source=3
)

Depth	Typical Pages	Time
1	10-50	Seconds
2	50-500	Minutes
3	500-5000	Many minutes

# Conservative (start here)
response = client.crawl(
    url="https://example.com",
    max_depth=1,
    max_breadth=20,
    limit=20
)

# Comprehensive (use carefully)
response = client.crawl(
    url="https://example.com",
    max_depth=3,
    max_breadth=100,
    limit=500
)

Problem	Impact	Solution
Excessive depth (`max_depth=4+`)	Exponential time, unnecessary pages	Start with 1-2, increase if needed
Unfocused crawling	Wasted resources, irrelevant content, context explosion	Use `instructions` to focus semantically
Missing limits	Runaway crawls, unexpected costs	Always set reasonable `limit` value
Ignoring `failed_results`	Incomplete data, missed content	Monitor and adjust parameters
Full content without chunks	Context window explosion	Use `instructions` + `chunks_per_source`

Field	Description
`base_url`	The URL you started the crawl from
`results`	List of crawled pages
`results[].url`	Page URL
`results[].raw_content`	Extracted content (or chunks if instructions provided)
`results[].images`	Image URLs extracted from the page
`results[].favicon`	Favicon URL (if `include_favicon=True`)
`response_time`	Time in seconds
`request_id`	Unique identifier for support reference

Crawl

Crawl & Map API Reference

Table of Contents

Crawl

Crawl & Map API Reference

Table of Contents

Crawl vs Map

Key Parameters

Instructions and Chunks

Path and Domain Filtering

Path patterns (regex)

Domain control (regex)

Use Cases

1. Deep/Unlinked Content

2. Documentation/Structured Content

3. Multi-modal/Cross-referencing

4. Rapidly Changing Content

5. RAG/Knowledge Base Integration

6. Compliance/Auditing

7. Known URL Patterns

Map then Extract Pattern

Performance Optimization

Depth vs Performance

Rate Limiting

Conservative vs Comprehensive

Common Pitfalls

Response Fields

Crawl Response

Map Response

Summary

Field	Description
`base_url`	The URL you started the mapping from
`results`	List of discovered URLs
`response_time`	Time in seconds
`request_id`	Unique identifier for support reference