MCPInstall

web-fetch

by alleneubankDocs & Writing

Fetches web content as clean markdown by preferring markdown-native responses and falling back to selector-based HTML extraction. Use for documentation, articles, and reference pages at http/https URLs.

Installs
978
Last 24h
—
Trending rank
—
Security
Not audited

Install

Works with every agent the skills CLI supports.

npx skills add https://github.com/alleneubank/claude-code --skill web-fetch
  1. 01

    Install

    Run the command in your project. The CLI asks which agents to install for.

  2. 02

    Describe the task

    Ask your agent for something this skill covers. It loads web-fetch automatically when relevant.

  3. 03

    Review the result

    The agent follows the SKILL.md steps, scripts and references bundled with the skill.

SKILL.md

The instructions your agent reads when the skill is triggered.

Web Content Fetching

Fetch web content in this order:

  1. Prefer markdown-native endpoints (content-type: text/markdown)
  2. Use selector-based HTML extraction for known sites
  3. Use the bundled Bun fallback script when selectors fail

Prerequisites

Verify required tools before extracting:

command -v curl >/dev/null || echo "curl is required"
command -v html2markdown >/dev/null || echo "html2markdown is required for HTML extraction"
command -v bun >/dev/null || echo "bun is required for fetch.ts fallback"

Install Bun dependencies for the bundled script:

cd ~/.claude/skills/web-fetch && bun install

Default Workflow

Use this as the default flow for any URL:

URL="<url>"
CONTENT_TYPE="$(curl -sIL "$URL" | awk -F': ' 'tolower($1)=="content-type"{print tolower($2)}' | tr -d '\r' | tail -1)"

if echo "$CONTENT_TYPE" | grep -q "markdown"; then
  curl -sL "$URL"
else
  curl -sL "$URL" \
    | html2markdown \
        --include-selector "article,main,[role=main]" \
        --exclude-selector "nav,header,footer,script,style"
fi

Known Site Selectors

SiteInclude SelectorExclude Selector
platform.claude.com#content-container-
docs.anthropic.com#content-container-
developer.mozilla.orgarticle-
github.com (docs)articlenav,.sidebar
Genericarticle,main,[role=main]nav,header,footer,script,style

Example:

curl -sL "<url>" \
  | html2markdown \
      --include-selector "#content-container" \
      --exclude-selector "nav,header,footer"

Finding the Right Selector

When a site isn't in the patterns list:

# Check what content containers exist
curl -s "<url>" | grep -o '<article[^>]*>\|<main[^>]*>\|id="[^"]*content[^"]*"' | head -10

# Test a selector
curl -sL "<url>" | html2markdown --include-selector "<selector>" | head -30

# Check line count
curl -sL "<url>" | html2markdown --include-selector "<selector>" | wc -l

Universal Fallback Script

When selectors produce poor output, run the bundled parser:

bun ~/.claude/skills/web-fetch/fetch.ts "<url>"

If already in the skill directory:

bun fetch.ts "<url>"

Options Reference

--include-selector "CSS"  # Keep only matching elements
--exclude-selector "CSS"  # Remove matching elements
--domain "https://..."    # Convert relative links to absolute

Troubleshooting

Empty output with selectors: The page might be markdown-native. Check headers first:

curl -sIL "<url>" | grep -i '^content-type:'

Wrong content selected: The site may have multiple article/main regions:

curl -s "<url>" | grep -o '<article[^>]*>'

html2markdown not found: Install it, then retry selector-based extraction.

bun or script deps missing: Run cd ~/.claude/skills/web-fetch && bun install.

Missing code blocks: Check if the site uses non-standard code formatting.

Client-rendered content: If HTML only has "Loading..." placeholders, the content is JS-rendered. Neither curl nor the Bun script can extract it; use browser-based tools.

FAQ

What does the web-fetch skill do?

Fetches web content as clean markdown by preferring markdown-native responses and falling back to selector-based HTML extraction. Use for documentation, articles, and reference pages at http/https URLs.

How do I install web-fetch?

Run npx skills add https://github.com/alleneubank/claude-code --skill web-fetch in your project. The skills CLI detects the agents you use and copies the skill into the right folder. For Claude Code you can also place the skill folder at ~/.claude/skills/web-fetch/ (personal) or .claude/skills/web-fetch/ (project).

Which AI agents can use web-fetch?

Any agent that supports the open Agent Skills format (a folder with a SKILL.md file), including Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf, Cline, OpenCode and more.

Is web-fetch safe to install?

No partner security audit is available yet. Skills can run commands and scripts, so read the SKILL.md and any bundled files before installing.

Who maintains web-fetch?

It is published by alleneubank in alleneubank/claude-code. It has 978 recorded installs.

More from alleneubank/claude-code

View all →

typescript-best-practices

alleneubank/claude-code

Use when reading or writing TypeScript or JavaScript files (.ts, .tsx, .js, tsconfig.json).

Code Quality & Workflow3.4K installs

react-best-practices

alleneubank/claude-code

Use when reading or writing React components (.tsx, .jsx files with React imports).

Frontend & Design3.2K installs

python-best-practices

alleneubank/claude-code

Use when reading or writing Python files (.py, pyproject.toml, requirements.txt).

Code Quality & Workflow2.2K installs

nix-best-practices

alleneubank/claude-code

Use when working with Nix flakes, overlays, shell.nix, or flake.nix files.

Code Quality & Workflow547 installs

zig-best-practices

alleneubank/claude-code

Use when reading or writing Zig files (.zig, build.zig, build.zig.zon).

Code Quality & Workflow524 installs

playwright-best-practices

alleneubank/claude-code

Open the skill to read its SKILL.md instructions.

Testing & QA373 installs

Popular docs & writing skills

View category →

grill-with-docs

mattpocock/skills

A relentless interview to sharpen a plan or design, which also creates docs (ADR's and glossary) as we go.

Docs & Writing1.2M installs

lark-markdown

open.feishu.cn

飞书 Markdown：查看、创建、上传、编辑和比较飞书中的原生 Markdown 文件。当用户要操作飞书 Markdown 文件，或比较其远端版本及本地草稿时使用。纯本地 Markdown 文件操作不触发本 skill。不负责将 Markdown 导入为飞书在线文档，也不负责文件搜索、权限、评论、移动、删除等云空间管理操作。

Docs & Writing778.7K installs

writing-beats

mattpocock/skills

Writing, exploit; assemble raw material into a journey of beats, grounding each term before a beat leans on it.

Docs & Writing490.7K installs

writing-shape

mattpocock/skills

Writing, exploit: shape raw material into an article, paragraph by paragraph.

Docs & Writing490.2K installs

writing-fragments

mattpocock/skills

Writing, explore: mine raw fragments, no structure yet.

Docs & Writing489.6K installs

writing-for-agents

mattpocock/skills

Writing documents for agents. Use when creating or editing skills, or modifying AGENTS.md or CLAUDE.md.

Docs & Writing479.3K installs