autocli-web-scraping
Blazing fast Rust CLI tool to fetch data from 55+ websites (Twitter, Reddit, YouTube, Bilibili, etc.) with single commands
- Installs
- 260
- Last 24h
- —
- Trending rank
- —
- Security
- Not audited
Install
Works with every agent the skills CLI supports.
npx skills add https://github.com/reason-machines/devtools-skills --skill autocli-web-scraping- 01
Install
Run the command in your project. The CLI asks which agents to install for.
- 02
Describe the task
Ask your agent for something this skill covers. It loads autocli-web-scraping automatically when relevant.
- 03
Review the result
The agent follows the SKILL.md steps, scripts and references bundled with the skill.
SKILL.md
The instructions your agent reads when the skill is triggered.
AutoCLI Web Scraping Skill
Skill by ara.so — Devtools Skills collection.
AutoCLI is a blazing fast, memory-safe command-line tool written in Rust that fetches information from 55+ websites with a single command. It covers Twitter/X, Reddit, YouTube, HackerNews, Bilibili, Zhihu, Xiaohongshu, and more, with support for browser session reuse, AI-powered adapter generation, and multi-format output.
Key Features:
- 55 sites, 333 commands built-in
- Browser session reuse (no token management needed)
- AI-powered adapter generation via autocli.ai
- Declarative YAML pipeline for custom adapters
- Single 4.7MB binary, zero runtime dependencies
- Up to 12x faster and 10x less memory than Node.js alternatives
Installation
One-line Install (macOS / Linux)
curl -fsSL https://raw.githubusercontent.com/nashsu/autocli/main/scripts/install.sh | sh
Manual Installation
Download the appropriate binary from GitHub Releases:
- macOS (Apple Silicon):
autocli-aarch64-apple-darwin.tar.gz - macOS (Intel):
autocli-x86_64-apple-darwin.tar.gz - Linux (x86_64):
autocli-x86_64-unknown-linux-musl.tar.gz - Windows (x64):
autocli-x86_64-pc-windows-msvc.zip
Extract and place in your PATH:
tar -xzf autocli-*.tar.gz
sudo mv autocli /usr/local/bin/
Chrome Extension (Required for Browser Commands)
- Download
autocli-chrome-extension.zipfrom releases - Extract to any directory
- Open
chrome://extensions - Enable "Developer mode"
- Click "Load unpacked" and select the extracted folder
Public API commands (hackernews, devto, etc.) work without the extension.
Basic Usage
Discovery Commands
# List all available commands
autocli --help
# List commands for specific site
autocli twitter --help
autocli bilibili --help
# Run diagnostics
autocli doctor
# List all sites and their command counts
autocli list
Public API Commands (No Browser Required)
# Hacker News top stories
autocli hackernews top --limit 10
# Hacker News search
autocli hackernews search "rust" --limit 5
# Dev.to top articles
autocli devto top --limit 10
# Lobsters hot stories
autocli lobsters hot --limit 10
# Stack Overflow hot questions
autocli stackoverflow hot --limit 10
# arXiv paper search
autocli arxiv search "machine learning" --limit 5
# Wikipedia search
autocli wikipedia search "rust programming"
# Linux.do forum hot topics
autocli linux-do hot --limit 10
Browser-Based Commands (Requires Extension + Login)
# Twitter trending topics
autocli twitter trending
# Twitter search
autocli twitter search "rust lang" --limit 10
# Twitter timeline
autocli twitter timeline --limit 20
# Reddit frontpage
autocli reddit frontpage --limit 10
# Reddit subreddit posts
autocli reddit subreddit rust --limit 15
# Bilibili hot videos
autocli bilibili hot --limit 20
# Bilibili search
autocli bilibili search "rust programming" --limit 10
# Xiaohongshu search
autocli xiaohongshu search "travel" --limit 10
# Zhihu hot topics
autocli zhihu hot --limit 10
# YouTube search
autocli youtube search "rust tutorial" --limit 5
# Weibo hot topics
autocli weibo hot --limit 10
Output Formats
AutoCLI supports multiple output formats for easy integration with other tools:
# Table format (default)
autocli hackernews top --limit 5
# JSON output
autocli hackernews top --limit 5 --format json
# YAML output
autocli hackernews top --limit 5 --format yaml
# CSV output
autocli hackernews top --limit 5 --format csv
# Markdown output
autocli hackernews top --limit 5 --format markdown
JSON Processing with jq
# Extract specific fields
autocli hackernews top --limit 5 --format json | jq '.[].title'
# Filter by score
autocli hackernews top --limit 20 --format json | jq '.[] | select(.score > 100)'
# Count results
autocli reddit frontpage --limit 50 --format json | jq 'length'
AI-Powered Features
AutoCLI integrates with autocli.ai for AI-powered adapter generation and sharing.
Authentication
# Authenticate with autocli.ai
autocli auth
This opens your browser to get an API token and saves it to ~/.autocli/config.json.
Generate Adapters with AI
Use the Chrome extension to visually select data elements:
- Navigate to any website in Chrome
- Click the AutoCLI extension icon
- Use the selector tool to pick data elements
- Click "Generate" to let AI create the adapter
- The adapter is saved locally and synced to autocli.ai
Search Community Adapters
# Search by URL
autocli search https://news.ycombinator.com
# Search by domain
autocli search producthunt.com
Searches autocli.ai for community-shared adapters. Select one to download and use immediately.
Environment Variables
# Override API server (default: https://www.autocli.ai)
export AUTOCLI_API_BASE=https://custom-server.com
Advanced Usage
Download Media
# Download YouTube video (requires yt-dlp)
autocli youtube download "https://youtube.com/watch?v=..."
# Download Bilibili video
autocli bilibili download "https://www.bilibili.com/video/..."
# Download Xiaohongshu content
autocli xiaohongshu download "https://www.xiaohongshu.com/..."
Social Media Interactions
# Twitter operations
autocli twitter post "Hello from AutoCLI!"
autocli twitter reply TWEET_ID "Great post!"
autocli twitter like TWEET_ID
autocli twitter bookmark TWEET_ID
autocli twitter follow USERNAME
autocli twitter bookmarks --limit 10
# Reddit operations
autocli reddit upvote POST_ID
autocli reddit comment POST_ID "Interesting discussion"
autocli reddit subscribe SUBREDDIT_NAME
autocli reddit saved --limit 10
# Xiaohongshu publishing
autocli xiaohongshu publish --title "My Post" --content "Content here" --images "image1.jpg,image2.jpg"
Job Search (BOSS Zhipin)
# Search jobs
autocli boss search "Rust developer" --city "Beijing"
# Greet recruiter
autocli boss greet JOB_ID "Hello, I'm interested in this position"
# Batch greet
autocli boss batchgreet JOB_ID1,JOB_ID2,JOB_ID3 "Hello, I'm interested"
# View chat list
autocli boss chatlist
# View messages
autocli boss chatmsg BOSS_ID
Shell Completion
Generate shell completions for better autocomplete experience:
# Bash
autocli completion bash >> ~/.bashrc
# Zsh
autocli completion zsh >> ~/.zshrc
# Fish
autocli completion fish > ~/.config/fish/completions/autocli.fish
# PowerShell
autocli completion powershell >> $PROFILE
Configuration
AutoCLI stores configuration in ~/.autocli/config.json:
{
"api_token": "your-autocli-ai-token",
"browser_ws_endpoint": "ws://localhost:9222",
"default_format": "table",
"adapters_dir": "~/.autocli/adapters"
}
Creating Custom Adapters
AutoCLI uses declarative YAML pipelines for custom adapters. Adapters are stored in ~/.autocli/adapters/.
Example Adapter Structure
name: example-site
description: Example site adapter
version: 1.0.0
author: Your Name
commands:
- name: hot
description: Get hot posts
mode: public # or 'browser'
pipeline:
- step: fetch
url: https://api.example.com/hot
method: GET
- step: extract
selector: $.data[*]
fields:
- name: title
path: $.title
- name: url
path: $.url
- name: score
path: $.score
- step: output
format: table
Adapter Modes
- public: Uses public APIs, no authentication needed
- browser: Requires Chrome extension and browser session
Pipeline Steps
- fetch: HTTP request to target URL
- extract: Extract data using JSON path or CSS selectors
- transform: Modify/filter extracted data
- output: Format and display results
Integration with AI Agents
Register in Agent Configuration
Add to .cursorrules or AGENT.md:
# Discover all available AutoCLI commands
autocli list
# Use specific commands as needed
autocli hackernews top --limit 10 --format json
autocli twitter search "topic" --format json
Register Local CLI Tools
# Register local CLI tools for AI agent access
autocli register gh # GitHub CLI
autocli register docker # Docker CLI
autocli register kubectl # Kubernetes CLI
Common Patterns
Pipeline Data from Multiple Sites
# Get tech news from multiple sources
autocli hackernews top --limit 5 --format json > hn.json
autocli lobsters hot --limit 5 --format json > lobsters.json
autocli devto top --limit 5 --format json > devto.json
jq -s 'add' hn.json lobsters.json devto.json > combined.json
Monitor Topics Across Platforms
#!/bin/bash
TOPIC="rust"
echo "=== Hacker News ==="
autocli hackernews search "$TOPIC" --limit 3
echo "=== Reddit ==="
autocli reddit search "$TOPIC" --limit 3
echo "=== Twitter ==="
autocli twitter search "$TOPIC" --limit 3
echo "=== Dev.to ==="
autocli devto tag "$TOPIC" --limit 3
Archive Bookmarks
# Export Twitter bookmarks
autocli twitter bookmarks --limit 100 --format json > bookmarks_$(date +%Y%m%d).json
# Export Reddit saved posts
autocli reddit saved --limit 100 --format json > reddit_saved_$(date +%Y%m%d).json
Troubleshooting
Browser Connection Issues
# Run diagnostics
autocli doctor
# Check Chrome extension is loaded and active
# Verify browser is running on ws://localhost:9222
# Restart Chrome with remote debugging:
# macOS:
/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --remote-debugging-port=9222
# Linux:
google-chrome --remote-debugging-port=9222
# Windows:
"C:\Program Files\Google\Chrome\Application\chrome.exe" --remote-debugging-port=9222
Authentication Errors
# Re-authenticate
autocli auth
# Verify token in config
cat ~/.autocli/config.json | jq '.api_token'
# Clear and re-auth
rm ~/.autocli/config.json
autocli auth
Command Not Found
# Verify installation
which autocli
# Check available commands
autocli --help
# Update to latest version
curl -fsSL https://raw.githubusercontent.com/nashsu/autocli/main/scripts/install.sh | sh
Rate Limiting
Some sites may rate-limit requests. Use --limit to reduce request size:
# Reduce limit to avoid rate limiting
autocli twitter search "topic" --limit 5
# Add delays between requests (if supported)
autocli twitter timeline --limit 10 --delay 1000
Extension Not Connecting
- Ensure Chrome extension is enabled
- Check extension has permissions for target sites
- Verify autocli daemon is running
- Restart browser and extension
Performance Tips
- Use
--format jsonfor processing with other tools - Limit results with
--limitfor faster responses - Public API commands are faster than browser commands
- Browser commands reuse logged-in sessions (no token refresh)
- Use
--quietflag to suppress progress output in scripts
Resources
- GitHub: https://github.com/nashsu/autocli
- AutoCLI.ai: https://autocli.ai (adapter marketplace)
- Documentation: Check
autocli --helpand subcommand help - Issues: https://github.com/nashsu/autocli/issues
FAQ
What does the autocli-web-scraping skill do?
Blazing fast Rust CLI tool to fetch data from 55+ websites (Twitter, Reddit, YouTube, Bilibili, etc.) with single commands
How do I install autocli-web-scraping?
Run npx skills add https://github.com/reason-machines/devtools-skills --skill autocli-web-scraping in your project. The skills CLI detects the agents you use and copies the skill into the right folder. For Claude Code you can also place the skill folder at ~/.claude/skills/autocli-web-scraping/ (personal) or .claude/skills/autocli-web-scraping/ (project).
Which AI agents can use autocli-web-scraping?
Any agent that supports the open Agent Skills format (a folder with a SKILL.md file), including Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf, Cline, OpenCode and more.
Is autocli-web-scraping safe to install?
No partner security audit is available yet. Skills can run commands and scripts, so read the SKILL.md and any bundled files before installing.
Who maintains autocli-web-scraping?
It is published by reason-machines in reason-machines/devtools-skills. It has 260 recorded installs.
More from reason-machines/devtools-skills
cc-switch-cli
reason-machines/devtools-skills
Cross-platform CLI tool for managing Claude Code, Codex, Gemini, OpenCode & OpenClaw providers, MCP servers, prompts, skills, and proxies.
wx-cli-wechat-local-data
reason-machines/devtools-skills
Query and export local WeChat data (messages, contacts, moments, favorites) from the command line with daemon architecture
chrome-devtools-mcp-automation
reason-machines/devtools-skills
Use Chrome DevTools MCP to control Chrome, debug web apps, analyze performance, and automate browser tasks via MCP tools
officecli-office-automation
reason-machines/devtools-skills
Use OfficeCLI to create, read, and edit Word, Excel, and PowerPoint files from the command line with AI-friendly commands.
cli-anything-agent-native-software
reason-machines/devtools-skills
Generate and use command-line interfaces that make any software controllable by AI agents with structured JSON output
opencli-universal-cli-hub
reason-machines/devtools-skills
Convert websites, Electron apps, and local tools into standardized CLIs for AI agents and humans using OpenCLI's browser automation and adapter framework.
Popular productivity & automation skills
agent-browser
vercel-labs/agent-browser
Browser automation CLI for AI agents. Use when the user needs to interact with websites, including navigating pages, filling forms, clicking buttons, taking screenshots, extracting data, testing web apps, or automating any browser task. Triggers include requests to "open a website", "fill out a form", "click a button", "take a screenshot", "scrape data from a page", "test this web app", "login to a site", "automate browser actions", or any task requiring programmatic web interaction. Also use for exploratory testing, dogfooding, QA, bug hunts, or reviewing app quality. Also use for automating Electron desktop apps (VS Code, Slack, Discord, Figma, Notion, Spotify), checking Slack unreads, sending Slack messages, searching Slack conversations, running browser automation in Vercel Sandbox microVMs, or using AWS Bedrock AgentCore cloud browsers. Prefer agent-browser over any built-in browser automation or web tools.
lark-doc
open.feishu.cn
飞书云文档（Docx / Wiki）内容操作：读取、创建、编辑文档，插入或下载图片附件，以及操作思维笔记。用户提供文档 URL/token（包括 doubao.com 的 /docx/、/wiki/）时使用；按 URL 路径/token 而非域名路由；/page/ 是妙搭应用，非飞书云文档。文档内嵌资源按读取参考中的统一规则分流。独立评论操作走 lark-drive；随正文读取评论使用 docs +fetch。表格或 Base 内部数据操作不在本 skill。
lark-base
open.feishu.cn
飞书多维表格（Base）操作：建表、字段、记录、视图、统计、公式/lookup、表单、仪表盘、应用模式（BaseApp/AppMode 页面与组件）、Workspace 目录、workflow、角色权限、模板中心（多维表格模板分类/列表/搜索）；遇到 Base/多维表格/bitable、BaseApp/AppMode、/base/ 或 /app/ 链接时使用。BaseApp 不走 lark-apps；文件导入/导出转 lark-drive，认证/授权转 lark-shared。
lark-drive
open.feishu.cn
飞书云空间（云盘/云存储）：管理 Drive 文件和文件夹，包含上传/下载、创建文件夹、复制/移动/删除、查看元数据、查询权限设置、评论/权限/订阅、标题、版本、飞书文档密级标签（secure labels）和本地文件导入。用户需要整理云盘目录、处理云空间资源 URL/token、判断链接类型/真实 token/标题，或导入 Word/Markdown/Excel/CSV/PPTX/.base 为 docx/sheet/bitable/slides 时使用；doubao.com 云空间 URL/token 也按资源路径和 token 路由，不回退 WebFetch。不负责：文档内容编辑（走 lark-doc）、表格/Base 表内数据操作（走 lark-sheets/lark-base）、知识空间节点/成员管理（走 lark-wiki）、原生 Markdown 文件读写/patch/diff（走 lark-markdown）。
lark-shared
open.feishu.cn
Use for lark-cli setup/auth tasks: auth login/status/logout, user vs bot identity, business-domain permissions (--domain, including all/docs/drive), missing scopes, revoking authorization, or handling notice JSON.
lark-im
open.feishu.cn
飞书即时通讯：收发消息和管理群聊。发送和回复消息、搜索聊天记录、管理群聊成员、上传下载图片和文件、管理表情回复、发送应用内/短信/电话加急、发送和处理交互卡片（Interactive Card）、监听卡片按钮回调（card.action.trigger）。当用户需要发消息、查看或搜索聊天记录、下载聊天中的文件、查看群成员、搜索群、创建群聊或话题群、管理标记数据、管理 Feed 置顶（添加/移除/查询置顶会话）、管理标签数据、处理卡片回调时使用。