Web Content Fetcher

Expert knowledge for fetching and parsing web content, handling size limitations, JavaScript-rendered sites, and extracting clean article content from HTML.

Overview

This skill provides battle-tested strategies for fetching web content in Claude Code, addressing critical challenges:

1. WebFetch tool limitation: ~50KB max content size
2. Read tool limitation: 256KB max file size per call
3. JavaScript-rendered content: Twitter/X, React SPAs require special handling

Four-tier approach:

• Tier 1: WebFetch for small content (< 50KB)
• Tier 2: curl + Task agent for any size static content ⭐ DEFAULT
• Tier 3: Scripts for edge cases (EUC-JP encoding, complex HTML)
• Tier 4: Playwright for JavaScript-rendered sites (Twitter/X, React SPAs)

Quick Tool Limitations

Tiered Fetching Strategy

Tier 1: Small Content - WebFetch Direct

Use when: Simple articles, API responses, first attempt on unknown content

Quick example:

WebFetch tool with url and extraction prompt

If fails: "Prompt too long" error → Switch to Tier 2

Tier 2: Any Size Content - curl + Task Agent ⭐ RECOMMENDED

Use when: News articles, blog posts, WebFetch fails, any content of any size

Quick workflow:

# 1. Create directory
mkdir -p output/tasks/YYYYMMDD_taskname/original

# 2. Fetch HTML
curl -s "URL" > output/tasks/YYYYMMDD_taskname/original/raw_html.html

# 3. Extract with Task agent
# Prompt: "Read HTML at [path], extract article content, save to fetched_content.md"

Why default: Handles any size, AI-powered extraction, no script setup needed.

For detailed workflow: See references/workflows.md

Tier 3: Script-Based Extraction - Edge Cases Only

Use when: Task agent struggles with complex HTML or special encoding requirements (EUC-JP)

Available scripts:

• scripts/extract_article.js - Standard HTML with Mozilla Readability
• scripts/extract_eucjp.js - Japanese EUC-JP encoded sites (4gamer.net)

Quick example:

node .claude/skills/web-content-fetcher/scripts/extract_article.js raw_html.html > fetched_content.md

Note: Try Task agent (Tier 2) first. Scripts only when Task agent explicitly fails.

For script details: See references/advanced-fetching.md

Tier 4: JavaScript-Rendered Content - Playwright

Use when: Twitter/X, React SPAs, dynamic sites, static fetch returns empty/error

Quick example:

node .claude/skills/web-content-fetcher/scripts/fetch_js_content.js \
  "https://x.com/user/status/123456789" \
  --output fetched_content.md

Common sites: Twitter/X, Threads, Instagram, React/Vue/Angular SPAs, AJAX-loaded content

Image Extraction:

• Automatically extracts images if exist (Twitter/X, Threads, blogs, etc.)
• Captures image URLs with alt text from main content area
• Filters out small images (icons/logos) automatically
• Output includes structured Media section with URLs for downloading

Setup required (one-time):

cd .claude/skills/web-content-fetcher
pnpm add -D playwright --save-catalog-name=dev
pnpm exec playwright install chromium

For Playwright details: See references/advanced-fetching.md

Decision Tree

Use this flowchart to choose the right approach:

Need to fetch web content?
│
├─ JavaScript-rendered site (Twitter/X, React SPA, dynamic content)?
│  └─ Use Playwright script (Tier 4)
│     └─ See: references/advanced-fetching.md
│
├─ Size unknown or small expected content?
│  └─ Try WebFetch (Tier 1)
│     ├─ Success? → Done ✓
│     └─ "Prompt too long" error? → Use Tier 2
│
├─ Any other case (medium/large content, WebFetch failed)?
│  └─ Use curl + Task agent (Tier 2) ⭐ DEFAULT
│     ├─ Success? → Done ✓
│     └─ Task agent struggles? → Use Tier 3
│        └─ See: references/advanced-fetching.md
│
└─ Edge cases (Task agent fails, special encoding)?
   └─ Use curl + script (Tier 3)
      └─ See: references/advanced-fetching.md

Standard Workflow Quick Reference

Most common pattern (works for 90% of cases):

# 1. Setup
mkdir -p output/tasks/20260111_taskname/original

# 2. Fetch
curl -s "URL" > output/tasks/20260111_taskname/original/raw_html.html

# 3. Extract (Task agent prompt)
"Read HTML file at [path], extract main article content,
remove navigation/ads/sidebars/comments/footer,
save clean markdown to: [path]/fetched_content.md"

# 4. Use
Read: output/tasks/20260111_taskname/original/fetched_content.md

For detailed workflows and patterns: See references/workflows.md

Common Use Cases

Translation with images:

1. Fetch content (Tier 2 or Tier 4 for JS sites)
2. Extract to original/fetched_content.md with readable format (includes Media section with image URLs)
3. Download images using URLs from Media section
4. Pass to /translate command

Analysis:

1. Fetch content (Tier 2 or Tier 4)
2. Extract to original/fetched_content.md with readable format
3. Read and analyze

Social media posts (Twitter/X, Threads, Instagram):

1. Use Playwright (Tier 4) directly
2. Automatically extracts text, images, and metadata
3. Outputs to fetched_content.md with Media section in a readable format
4. Download images from extracted URLs
5. Use for translation/analysis

For all patterns: See references/workflows.md

When Things Go Wrong

Common issues and quick fixes:

For complete troubleshooting guide: See references/troubleshooting.md

Best Practices

1. Always use task directories: output/tasks/YYYYMMDD_taskname/
2. Default to Tier 2: curl + Task agent works for nearly all cases
3. Keep raw HTML: Save to original/raw_html.html for reference (except Tier 4)
4. Use Playwright for JS sites: Twitter/X, React SPAs, dynamic content
5. Clean content format: Always save extracted content as markdown
6. Descriptive naming: Use date prefix (YYYYMMDD_) and descriptive task names
7. Try simple first: WebFetch → curl + Task agent → Scripts (only if needed)
8. Task agent for extraction: Let AI handle pagination and parsing complexity

Reference Files

This skill uses progressive disclosure for efficiency. Core information is in this file. Detailed guides are in reference files:

• references/workflows.md - Standard workflows, common patterns, directory structure, content extraction priorities
• references/advanced-fetching.md - Tier 3 script-based extraction, Tier 4 Playwright details, setup instructions
• references/troubleshooting.md - Common issues, solutions, quick reference commands

Read reference files when you need detailed guidance for specific scenarios.

Quick Start Examples

Simple article:

mkdir -p output/tasks/20260111_article/original
curl -s "https://example.com/article" > output/tasks/20260111_article/original/raw_html.html
# Task agent: Extract to fetched_content.md

Twitter/X post:

mkdir -p output/tasks/20260111_twitter/original
node .claude/skills/web-content-fetcher/scripts/fetch_js_content.js \
  "https://x.com/user/status/123" \
  --output output/tasks/20260111_twitter/original/fetched_content.md

Threads post:

mkdir -p output/tasks/20260111_threads/original
node .claude/skills/web-content-fetcher/scripts/fetch_js_content.js \
  "https://www.threads.com/@user/post/ABC123" \
  --output output/tasks/20260111_threads/original/fetched_content.md

Output format with images (works for all sites):

# Title

Main content text here...

### Media

1. Image description or alt text
   - URL: https://example.com/image1.jpg
2. Another image
   - URL: https://example.com/image2.jpg

Note:

• Images are automatically extracted from the main content area
• Small images (< 100x100px) like icons/logos are filtered out
• Image formats: .jpg, .png, .webp, .gif, .svg - all preserved in URLs
• When downloading, respect the original file extension

Japanese site (potential encoding issue):

mkdir -p output/tasks/20260111_japanese/original
curl -s "https://4gamer.net/..." > output/tasks/20260111_japanese/original/raw_html.html
# Task agent with encoding detection, or use extract_eucjp.js if needed