web-scraper
Extract structured data from websites. Use when: collecting competitor pricing; scraping product listings; extracting contact information; gathering research data; monitoring website changes
- Installs
- 582
- Last 24h
- —
- Trending rank
- —
- Security
- Not audited
Install
Works with every agent the skills CLI supports.
npx skills add https://github.com/guia-matthieu/clawfu-skills --skill web-scraper- 01
Install
Run the command in your project. The CLI asks which agents to install for.
- 02
Describe the task
Ask your agent for something this skill covers. It loads web-scraper automatically when relevant.
- 03
Review the result
The agent follows the SKILL.md steps, scripts and references bundled with the skill.
SKILL.md
The instructions your agent reads when the skill is triggered.
Web Scraper
Extract structured data from websites using BeautifulSoup and requests - turn any webpage into usable data.
When to Use This Skill
- Competitor research - Scrape pricing, features, positioning
- Lead generation - Extract contact info from directories
- Content audit - Pull headings, links, meta data
- Price monitoring - Track competitor pricing changes
- Data collection - Gather research data from multiple sources
What Claude Does vs What You Decide
| Claude Does | You Decide |
|---|---|
| Structures analysis frameworks | Strategic priorities |
| Synthesizes market data | Competitive positioning |
| Identifies opportunities | Resource allocation |
| Creates strategic options | Final strategy selection |
| Suggests implementation approaches | Execution decisions |
Dependencies
pip install beautifulsoup4 requests pandas click lxml
Commands
Scrape Elements
python scripts/main.py scrape https://example.com --selector "h1,h2,p"
python scripts/main.py scrape https://example.com --selector ".product-price"
Extract Links
python scripts/main.py links https://example.com
python scripts/main.py links https://example.com --internal-only
Extract Emails
python scripts/main.py emails https://example.com
python scripts/main.py emails https://example.com --depth 2
Extract Structured Data
python scripts/main.py structured https://example.com/article --schema article
python scripts/main.py structured https://example.com/product --schema product
Examples
Example 1: Scrape Competitor Pricing
python scripts/main.py scrape https://competitor.com/pricing --selector ".price,.plan-name"
# Output:
# Extracted 6 elements
# 1. Starter - $29/mo
# 2. Pro - $99/mo
# 3. Enterprise - Contact us
Example 2: Extract Article Content
python scripts/main.py structured https://blog.example.com/post --schema article
# Output: article_data.json
# {
# "title": "How to Scale Your Startup",
# "author": "Jane Doe",
# "date": "2024-01-15",
# "content": "...",
# "word_count": 1523
# }
CSS Selector Reference
| Selector | Description | Example |
|---|---|---|
tag | Element type | h1, p, div |
.class | Class name | .price, .title |
#id | Element ID | #main-content |
tag.class | Tag with class | div.product |
tag[attr] | Has attribute | a[href] |
parent > child | Direct child | ul > li |
tag1, tag2 | Multiple | h1, h2, h3 |
Ethical Scraping Guidelines
- Check robots.txt - Respect site's scraping policy
- Rate limit - Don't overload servers (1-2 req/sec)
- Identify yourself - Use descriptive User-Agent
- Cache requests - Don't re-scrape unchanged pages
- Terms of Service - Check if scraping is allowed
Skill Boundaries
What This Skill Does Well
- Structuring strategic analysis
- Identifying market opportunities
- Creating strategic frameworks
- Synthesizing competitive data
What This Skill Cannot Do
- Replace market research
- Guarantee strategic success
- Know proprietary competitor info
- Make executive decisions
Related Skills
- competitor-monitor - Monitor competitor changes
- pdf-extractor - Extract from PDFs
Skill Metadata
- Mode: centaur
category: automation
subcategory: data-extraction
dependencies: [beautifulsoup4, requests, pandas]
difficulty: intermediate
time_saved: 5+ hours/week
FAQ
What does the web-scraper skill do?
Extract structured data from websites. Use when: collecting competitor pricing; scraping product listings; extracting contact information; gathering research data; monitoring website changes
How do I install web-scraper?
Run npx skills add https://github.com/guia-matthieu/clawfu-skills --skill web-scraper in your project. The skills CLI detects the agents you use and copies the skill into the right folder. For Claude Code you can also place the skill folder at ~/.claude/skills/web-scraper/ (personal) or .claude/skills/web-scraper/ (project).
Which AI agents can use web-scraper?
Any agent that supports the open Agent Skills format (a folder with a SKILL.md file), including Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf, Cline, OpenCode and more.
Is web-scraper safe to install?
No partner security audit is available yet. Skills can run commands and scripts, so read the SKILL.md and any bundled files before installing.
Who maintains web-scraper?
It is published by guia-matthieu in guia-matthieu/clawfu-skills. It has 582 recorded installs.
More from guia-matthieu/clawfu-skills
whisper-transcription
guia-matthieu/clawfu-skills
Transcribe audio and video files to text using OpenAI Whisper. Use when: converting podcasts to blog posts; creating video subtitles; extracting quotes from interviews; repurposing video content to text; building searchable audio archives
yc-pitch-deck
guia-matthieu/clawfu-skills
Create a compelling startup pitch deck that follows the structure proven to raise billions from top investors. Master the YC and Sequoia formats that get founders funded. Use when: Fundraising to create investor pitch decks; YC application to structure your narrative; Demo Day prep to craft your 2-minute pitch; Angel investors to communicate your opportunity; Partnership pitches to structure compelling asks
conversion-copywriting
guia-matthieu/clawfu-skills
Write copy that gets a "yes" using Joanna Wiebe's research-first, Voice of Customer methodology Use when: Writing landing pages, emails, or sales pages that need measurable conversion results; Starting a new copy project and need a systematic process to follow; Struggling with what to write and staring at a blank page; Wanting to prove ROI to clients with data-backed decisions; Improving existing copy through validation and testing
sound-design-film
guia-matthieu/clawfu-skills
Apply Walter Murch's legendary film sound principles to marketing video, creating emotionally resonant audio that audiences feel without consciously noticing. Use when: Designing sound for brand films and documentaries; Creating emotional impact in video ads; Layering music, effects, and voice effectively; Building immersive soundscapes for content; Elevating production value through audio
report-generator
guia-matthieu/clawfu-skills
Generate PDF/HTML reports from templates and data. Use when: creating client reports; generating weekly summaries; producing marketing performance reports; automating recurring reports
persuasion-principles
guia-matthieu/clawfu-skills
Master Robert Cialdini's 6 (+1) Principles of Persuasion from "Influence: The Psychology of Persuasion" (1984). Ethically apply the psychology of compliance to marketing. Use when: Designing landing pages that convert; Writing sales copy and email sequences; Creating pricing and offer structures; Building testimonial and social proof strategies; Developing referral and loyalty programs
Popular productivity & automation skills
agent-browser
vercel-labs/agent-browser
Browser automation CLI for AI agents. Use when the user needs to interact with websites, including navigating pages, filling forms, clicking buttons, taking screenshots, extracting data, testing web apps, or automating any browser task. Triggers include requests to "open a website", "fill out a form", "click a button", "take a screenshot", "scrape data from a page", "test this web app", "login to a site", "automate browser actions", or any task requiring programmatic web interaction. Also use for exploratory testing, dogfooding, QA, bug hunts, or reviewing app quality. Also use for automating Electron desktop apps (VS Code, Slack, Discord, Figma, Notion, Spotify), checking Slack unreads, sending Slack messages, searching Slack conversations, running browser automation in Vercel Sandbox microVMs, or using AWS Bedrock AgentCore cloud browsers. Prefer agent-browser over any built-in browser automation or web tools.
lark-doc
open.feishu.cn
飞书云文档（Docx / Wiki）内容操作：读取、创建、编辑文档，插入或下载图片附件，以及操作思维笔记。用户提供文档 URL/token（包括 doubao.com 的 /docx/、/wiki/）时使用；按 URL 路径/token 而非域名路由；/page/ 是妙搭应用，非飞书云文档。文档内嵌资源按读取参考中的统一规则分流。独立评论操作走 lark-drive；随正文读取评论使用 docs +fetch。表格或 Base 内部数据操作不在本 skill。
lark-base
open.feishu.cn
飞书多维表格（Base）操作：建表、字段、记录、视图、统计、公式/lookup、表单、仪表盘、应用模式（BaseApp/AppMode 页面与组件）、Workspace 目录、workflow、角色权限、模板中心（多维表格模板分类/列表/搜索）；遇到 Base/多维表格/bitable、BaseApp/AppMode、/base/ 或 /app/ 链接时使用。BaseApp 不走 lark-apps；文件导入/导出转 lark-drive，认证/授权转 lark-shared。
lark-drive
open.feishu.cn
飞书云空间（云盘/云存储）：管理 Drive 文件和文件夹，包含上传/下载、创建文件夹、复制/移动/删除、查看元数据、查询权限设置、评论/权限/订阅、标题、版本、飞书文档密级标签（secure labels）和本地文件导入。用户需要整理云盘目录、处理云空间资源 URL/token、判断链接类型/真实 token/标题，或导入 Word/Markdown/Excel/CSV/PPTX/.base 为 docx/sheet/bitable/slides 时使用；doubao.com 云空间 URL/token 也按资源路径和 token 路由，不回退 WebFetch。不负责：文档内容编辑（走 lark-doc）、表格/Base 表内数据操作（走 lark-sheets/lark-base）、知识空间节点/成员管理（走 lark-wiki）、原生 Markdown 文件读写/patch/diff（走 lark-markdown）。
lark-shared
open.feishu.cn
Use for lark-cli setup/auth tasks: auth login/status/logout, user vs bot identity, business-domain permissions (--domain, including all/docs/drive), missing scopes, revoking authorization, or handling notice JSON.
lark-im
open.feishu.cn
飞书即时通讯：收发消息和管理群聊。发送和回复消息、搜索聊天记录、管理群聊成员、上传下载图片和文件、管理表情回复、发送应用内/短信/电话加急、发送和处理交互卡片（Interactive Card）、监听卡片按钮回调（card.action.trigger）。当用户需要发消息、查看或搜索聊天记录、下载聊天中的文件、查看群成员、搜索群、创建群聊或话题群、管理标记数据、管理 Feed 置顶（添加/移除/查询置顶会话）、管理标签数据、处理卡片回调时使用。