MCPInstall

web-scraping-automation

by aaaaqwqProductivity & Automation

自动化爬取网站数据和 API 接口。当用户需要抓取网页内容、调用 API、解析数据或创建爬虫脚本时使用此技能。

Installs
630
Last 24h
—
Trending rank
—
Security
Not audited

Install

Works with every agent the skills CLI supports.

npx skills add https://github.com/aaaaqwq/agi-super-team --skill web-scraping-automation
  1. 01

    Install

    Run the command in your project. The CLI asks which agents to install for.

  2. 02

    Describe the task

    Ask your agent for something this skill covers. It loads web-scraping-automation automatically when relevant.

  3. 03

    Review the result

    The agent follows the SKILL.md steps, scripts and references bundled with the skill.

SKILL.md

The instructions your agent reads when the skill is triggered.

网站爬取与 API 自动化

功能说明

此技能专门用于自动化网站数据爬取和 API 接口调用，包括：

  • 分析和爬取网站结构
  • 调用和测试 REST/GraphQL API
  • 创建自动化爬虫脚本
  • 数据解析和清洗
  • 处理反爬虫机制
  • 定时任务和数据存储

使用场景

  • "爬取这个网站的产品信息"
  • "帮我调用这个 API 并解析返回数据"
  • "创建一个脚本定时抓取新闻"
  • "分析这个网站的 API 接口文档"
  • "绕过这个网站的反爬虫限制"

技术栈

⚠️ 资源清理原则（强制）

所有涉及浏览器的爬取任务完成后，必须自动关闭 Chrome/Selenium 进程！

# Playwright 示例
from playwright.sync_api import sync_playwright

def scrape_website():
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page = browser.new_page()
        # ... 爬取逻辑 ...
        browser.close()

    # ⚠️ 强制清理残留进程
    import subprocess
    subprocess.run(['pkill', '-f', 'chrome'], capture_output=True)

# Selenium 示例
from selenium import webdriver

driver = webdriver.Chrome()
try:
    # ... 爬取逻辑 ...
    pass
finally:
    driver.quit()
    # ⚠️ 确保清理
    import subprocess
    subprocess.run(['pkill', '-f', 'chrome'], capture_output=True)

原因: 避免内存泄漏和资源占用，防止 Gateway CPU 100% 过载

Python 爬虫

  • requests：HTTP 请求库
  • BeautifulSoup4：HTML 解析
  • Scrapy：专业爬虫框架
  • Selenium：浏览器自动化
  • Playwright：现代浏览器自动化

JavaScript 爬虫

  • axios：HTTP 客户端
  • cheerio：服务端 jQuery
  • puppeteer：Chrome 自动化
  • node-fetch：Fetch API

工作流程

  1. 目标分析：

    • 检查网站结构和数据位置
    • 分析 API 接口和认证方式
    • 评估反爬虫机制

  2. 方案设计：

    • 选择合适的技术栈
    • 设计数据提取策略
    • 规划错误处理和重试机制

  3. 脚本开发：

    • 编写爬虫代码
    • 实现数据解析逻辑
    • 添加日志和监控

  4. 测试优化：

    • 验证数据准确性
    • 优化性能和稳定性
    • 处理边界情况

最佳实践

  • 遵守 robots.txt 规则
  • 设置合理的请求间隔
  • 使用 User-Agent 和请求头
  • 实现错误重试机制
  • 数据去重和验证
  • 使用代理池（如需要）
  • 保存原始数据和日志

常见场景示例

1. 简单网页爬取

import requests
from bs4 import BeautifulSoup

def scrape_website(url):
    headers = {'User-Agent': 'Mozilla/5.0'}
    response = requests.get(url, headers=headers)
    soup = BeautifulSoup(response.text, 'html.parser')

    # 提取数据
    data = []
    for item in soup.select('.product'):
        data.append({
            'title': item.select_one('.title').text,
            'price': item.select_one('.price').text
        })
    return data

2. API 调用

import requests

def call_api(endpoint, params=None):
    headers = {
        'Authorization': 'Bearer YOUR_TOKEN',
        'Content-Type': 'application/json'
    }
    response = requests.get(endpoint, headers=headers, params=params)
    return response.json()

3. 动态网页爬取

from selenium import webdriver
from selenium.webdriver.common.by import By

def scrape_dynamic_page(url):
    driver = webdriver.Chrome()
    driver.get(url)

    # 等待页面加载
    driver.implicitly_wait(10)

    # 提取数据
    elements = driver.find_elements(By.CLASS_NAME, 'item')
    data = [elem.text for elem in elements]

    driver.quit()
    return data

反爬虫应对策略

  • 请求头伪装：模拟真实浏览器
  • 代理轮换：使用代理池
  • 验证码处理：OCR 或第三方服务
  • Cookie 管理：维护会话状态
  • 请求频率控制：避免触发限制
  • JavaScript 渲染：使用 Selenium/Playwright

数据存储方案

  • CSV/Excel：简单数据导出
  • JSON：结构化数据存储
  • 数据库：MySQL、PostgreSQL、MongoDB
  • 云存储：S3、OSS
  • 数据仓库：用于大规模数据分析

FAQ

What does the web-scraping-automation skill do?

自动化爬取网站数据和 API 接口。当用户需要抓取网页内容、调用 API、解析数据或创建爬虫脚本时使用此技能。

How do I install web-scraping-automation?

Run npx skills add https://github.com/aaaaqwq/agi-super-team --skill web-scraping-automation in your project. The skills CLI detects the agents you use and copies the skill into the right folder. For Claude Code you can also place the skill folder at ~/.claude/skills/web-scraping-automation/ (personal) or .claude/skills/web-scraping-automation/ (project).

Which AI agents can use web-scraping-automation?

Any agent that supports the open Agent Skills format (a folder with a SKILL.md file), including Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf, Cline, OpenCode and more.

Is web-scraping-automation safe to install?

No partner security audit is available yet. Skills can run commands and scripts, so read the SKILL.md and any bundled files before installing.

Who maintains web-scraping-automation?

It is published by aaaaqwq in aaaaqwq/agi-super-team. It has 630 recorded installs.

More from aaaaqwq/agi-super-team

View all →

multi-search-engine

aaaaqwq/agi-super-team

Multi search engine integration with 17 engines (8 CN + 9 Global). Supports advanced search operators, time filters, site search, privacy engines, and WolframAlpha knowledge queries. No API keys required.

Backend & APIs2.7K installs

wechat-channel

aaaaqwq/agi-super-team

微信 (WeChat) 与 OpenClaw 的双向集成通道。基于 Wechaty + PadLocal 实现微信消息的接收和发送，支持私聊、群聊、@提及检测、图片/文件传输。当需要通过微信与 AI 助手交互、接收微信消息触发 AI 响应、或从 OpenClaw 发送消息到微信时使用此技能。

522 installs

project-management

aaaaqwq/agi-super-team

Open the skill to read its SKILL.md instructions.

468 installs

wecom-automation

aaaaqwq/agi-super-team

Open the skill to read its SKILL.md instructions.

Productivity & Automation422 installs

feishu-automation

aaaaqwq/agi-super-team

Open the skill to read its SKILL.md instructions.

Productivity & Automation366 installs

memory-hygiene

aaaaqwq/agi-super-team

Open the skill to read its SKILL.md instructions.

363 installs

Popular productivity & automation skills

View category →

agent-browser

vercel-labs/agent-browser

Browser automation CLI for AI agents. Use when the user needs to interact with websites, including navigating pages, filling forms, clicking buttons, taking screenshots, extracting data, testing web apps, or automating any browser task. Triggers include requests to "open a website", "fill out a form", "click a button", "take a screenshot", "scrape data from a page", "test this web app", "login to a site", "automate browser actions", or any task requiring programmatic web interaction. Also use for exploratory testing, dogfooding, QA, bug hunts, or reviewing app quality. Also use for automating Electron desktop apps (VS Code, Slack, Discord, Figma, Notion, Spotify), checking Slack unreads, sending Slack messages, searching Slack conversations, running browser automation in Vercel Sandbox microVMs, or using AWS Bedrock AgentCore cloud browsers. Prefer agent-browser over any built-in browser automation or web tools.

OfficialProductivity & Automation1.3M installs

lark-doc

open.feishu.cn

飞书云文档（Docx / Wiki）内容操作：读取、创建、编辑文档，插入或下载图片附件，以及操作思维笔记。用户提供文档 URL/token（包括 doubao.com 的 /docx/、/wiki/）时使用；按 URL 路径/token 而非域名路由；/page/ 是妙搭应用，非飞书云文档。文档内嵌资源按读取参考中的统一规则分流。独立评论操作走 lark-drive；随正文读取评论使用 docs +fetch。表格或 Base 内部数据操作不在本 skill。

Productivity & Automation815.7K installs

lark-base

open.feishu.cn

飞书多维表格（Base）操作：建表、字段、记录、视图、统计、公式/lookup、表单、仪表盘、应用模式（BaseApp/AppMode 页面与组件）、Workspace 目录、workflow、角色权限、模板中心（多维表格模板分类/列表/搜索）；遇到 Base/多维表格/bitable、BaseApp/AppMode、/base/ 或 /app/ 链接时使用。BaseApp 不走 lark-apps；文件导入/导出转 lark-drive，认证/授权转 lark-shared。

Productivity & Automation815K installs

lark-drive

open.feishu.cn

飞书云空间（云盘/云存储）：管理 Drive 文件和文件夹，包含上传/下载、创建文件夹、复制/移动/删除、查看元数据、查询权限设置、评论/权限/订阅、标题、版本、飞书文档密级标签（secure labels）和本地文件导入。用户需要整理云盘目录、处理云空间资源 URL/token、判断链接类型/真实 token/标题，或导入 Word/Markdown/Excel/CSV/PPTX/.base 为 docx/sheet/bitable/slides 时使用；doubao.com 云空间 URL/token 也按资源路径和 token 路由，不回退 WebFetch。不负责：文档内容编辑（走 lark-doc）、表格/Base 表内数据操作（走 lark-sheets/lark-base）、知识空间节点/成员管理（走 lark-wiki）、原生 Markdown 文件读写/patch/diff（走 lark-markdown）。

Productivity & Automation815K installs

lark-shared

open.feishu.cn

Use for lark-cli setup/auth tasks: auth login/status/logout, user vs bot identity, business-domain permissions (--domain, including all/docs/drive), missing scopes, revoking authorization, or handling notice JSON.

Productivity & Automation814.8K installs

lark-im

open.feishu.cn

飞书即时通讯：收发消息和管理群聊。发送和回复消息、搜索聊天记录、管理群聊成员、上传下载图片和文件、管理表情回复、发送应用内/短信/电话加急、发送和处理交互卡片（Interactive Card）、监听卡片按钮回调（card.action.trigger）。当用户需要发消息、查看或搜索聊天记录、下载聊天中的文件、查看群成员、搜索群、创建群聊或话题群、管理标记数据、管理 Feed 置顶（添加/移除/查询置顶会话）、管理标签数据、处理卡片回调时使用。

Productivity & Automation814.1K installs