MCPInstall

agent-reach-internet-access

Give AI agents eyes to see the internet — scrape Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu with zero API fees

Installs
229
Last 24h
—
Trending rank
—
Security
Not audited

Install

Works with every agent the skills CLI supports.

npx skills add https://github.com/reason-machines/ai-agent-skills --skill agent-reach-internet-access
  1. 01

    Install

    Run the command in your project. The CLI asks which agents to install for.

  2. 02

    Describe the task

    Ask your agent for something this skill covers. It loads agent-reach-internet-access automatically when relevant.

  3. 03

    Review the result

    The agent follows the SKILL.md steps, scripts and references bundled with the skill.

SKILL.md

The instructions your agent reads when the skill is triggered.

Agent Reach — Internet Access for AI Agents

Skill by ara.so — AI Agent Skills collection.

Agent Reach is a scaffolding tool that gives AI agents the ability to read and search across Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu, and more — all without paid APIs. It orchestrates best-in-class upstream tools (yt-dlp, twitter-cli, rdt-cli, gh CLI, etc.) and provides a unified interface for AI agents.

Installation

Agent Reach is installed via pip and automatically sets up dependencies:

# Basic installation
pip install agent-reach

# The tool will auto-detect and install:
# - Node.js (for some MCP servers)
# - gh CLI (for GitHub)
# - mcporter (for MCP integrations)
# - twitter-cli (for Twitter/X)
# - rdt-cli (for Reddit)
# - yt-dlp (for YouTube/Bilibili)

After installation, run diagnostics to check what's working:

agent-reach doctor

This shows status for each channel: ✅ (works out of box), 🔧 (needs config), or ❌ (not available).

Core Capabilities

1. Web Page Reading (Jina Reader)

Read any web page as clean markdown:

# Read a web page
curl https://r.jina.ai/https://example.com

# Get JSON format
curl https://r.jina.ai/https://example.com \
  -H "Accept: application/json"

# With images
curl https://r.jina.ai/https://example.com \
  -H "X-With-Images-Summary: true"

Python usage:

import requests

url = "https://example.com"
response = requests.get(f"https://r.jina.ai/{url}")
markdown_content = response.text

# With options
headers = {
    "X-With-Links-Summary": "true",
    "X-With-Images-Summary": "true"
}
response = requests.get(f"https://r.jina.ai/{url}", headers=headers)

2. YouTube & Video (yt-dlp)

Extract subtitles, metadata, and search videos:

# Get video metadata + subtitles
yt-dlp --dump-json --write-auto-subs --skip-download \
  "https://www.youtube.com/watch?v=VIDEO_ID"

# Search YouTube
yt-dlp "ytsearch5:AI agents tutorial" --dump-json

# Get specific subtitle language
yt-dlp --write-subs --sub-lang en --skip-download URL

# Bilibili videos (works same way)
yt-dlp --dump-json "https://www.bilibili.com/video/BV..."

Python usage:

import subprocess
import json

def get_video_info(url):
    result = subprocess.run(
        ["yt-dlp", "--dump-json", "--write-auto-subs", 
         "--skip-download", url],
        capture_output=True, text=True
    )
    return json.loads(result.stdout)

# Search videos
def search_youtube(query, max_results=5):
    result = subprocess.run(
        ["yt-dlp", f"ytsearch{max_results}:{query}", "--dump-json"],
        capture_output=True, text=True
    )
    return [json.loads(line) for line in result.stdout.strip().split('\n')]

3. Twitter/X (twitter-cli)

Requires Cookie authentication. Export cookies using Cookie-Editor Chrome extension.

# Configure (paste exported cookies when prompted)
twitter configure

# Read a tweet
twitter tweet https://twitter.com/user/status/123456789

# Search tweets
twitter search "AI agents" --limit 20

# Get user timeline
twitter timeline @username --limit 50

# Get tweet thread
twitter thread https://twitter.com/user/status/123456789

Configuration file location: ~/.twitter-cli/config.json

4. Reddit (rdt-cli)

Requires Cookie authentication:

# Login with cookies
rdt login

# Search posts
rdt search "machine learning" --limit 20

# Read post with comments
rdt post https://reddit.com/r/programming/comments/...

# Get subreddit posts
rdt subreddit r/python --limit 30

5. GitHub (gh CLI)

# Login (opens browser OAuth flow)
gh auth login

# View repository
gh repo view owner/repo

# Search repositories
gh search repos "LLM framework" --limit 20

# Search issues
gh search issues "bug" --repo owner/repo

# View issue
gh issue view 123 --repo owner/repo

# Create issue
gh issue create --repo owner/repo \
  --title "Bug report" --body "Description"

Python usage:

import subprocess
import json

def search_repos(query, limit=20):
    result = subprocess.run(
        ["gh", "search", "repos", query, 
         "--limit", str(limit), "--json", "name,description,url"],
        capture_output=True, text=True
    )
    return json.loads(result.stdout)

def get_repo_info(owner_repo):
    result = subprocess.run(
        ["gh", "repo", "view", owner_repo, "--json", 
         "description,stargazerCount,forkCount,url"],
        capture_output=True, text=True
    )
    return json.loads(result.stdout)

6. XiaoHongShu (xhs-cli via mcporter)

Requires Cookie authentication:

# Configure (sets up MCP server)
mcporter add xiaohongshu

# The MCP server provides these tools:
# - search_notes: Search XHS posts
# - get_note_detail: Get post content
# - post_note: Create new post
# - comment_note: Add comment
# - like_note: Like a post

Configuration stored in: ~/.mcporter/xiaohongshu/config.json

7. Bilibili Enhanced (bili-cli)

# Get hot videos
bili hot --limit 20

# Search videos
bili search "Python tutorial" --limit 30

# Get video info
bili video BV1xx411c7mD

# Get user dynamics
bili user-dynamic 123456

8. Internet Search (Exa via mcporter)

Semantic search across the web:

# Add Exa MCP server (no API key needed for basic use)
mcporter add exa

# The MCP server provides:
# - search: AI-powered semantic search
# - find_similar: Find similar pages
# - get_contents: Extract page contents

For advanced features, set API key:

export EXA_API_KEY=your_key_here

9. RSS Feeds

import feedparser

# Parse RSS feed
feed = feedparser.parse("https://example.com/feed.xml")

for entry in feed.entries:
    print(f"Title: {entry.title}")
    print(f"Link: {entry.link}")
    print(f"Published: {entry.published}")
    print(f"Summary: {entry.summary}")
    print("---")

10. WeChat Official Accounts

Search and read WeChat articles via Exa + Camoufox:

# Use Exa search to find WeChat articles
# Articles are auto-extracted when URLs contain mp.weixin.qq.com

11. Weibo (微博)

# Search content
agent-reach weibo search "AI" --type content

# Get hot search
agent-reach weibo hot

# Get user posts
agent-reach weibo user USER_ID

# Get comments
agent-reach weibo comments POST_ID

12. V2EX

# Get hot topics
agent-reach v2ex hot

# Get node topics
agent-reach v2ex node python

# Get topic details
agent-reach v2ex topic 123456

Configuration Patterns

Cookie-Based Services

For Twitter, Reddit, XiaoHongShu — use Cookie-Editor:

  1. Login to the service in browser
  2. Install Cookie-Editor
  3. Click extension → Export → Copy
  4. Paste into CLI config command

Never commit cookies to version control. They're stored in:

  • Twitter: ~/.twitter-cli/config.json
  • Reddit: ~/.rdt-cli/cookies.json
  • XHS: ~/.mcporter/xiaohongshu/config.json

Proxy Configuration (Server Deployments)

For Bilibili access from servers:

# Set proxy environment variables
export HTTP_PROXY=http://proxy-server:port
export HTTPS_PROXY=http://proxy-server:port

# Or configure per-tool
yt-dlp --proxy http://proxy-server:port URL

GitHub Authentication

# OAuth login (recommended)
gh auth login

# Or use token
export GITHUB_TOKEN=ghp_your_token_here
gh auth login --with-token <<< $GITHUB_TOKEN

Common Workflows

Scrape Twitter Thread for Research

import subprocess
import json

def get_twitter_thread(url):
    result = subprocess.run(
        ["twitter", "thread", url],
        capture_output=True, text=True
    )
    return result.stdout

thread_content = get_twitter_thread(
    "https://twitter.com/user/status/123456789"
)

Extract YouTube Video Summary

import subprocess
import json

def get_video_transcript(url):
    # Get metadata + subtitles
    result = subprocess.run(
        ["yt-dlp", "--dump-json", "--write-auto-subs", 
         "--skip-download", url],
        capture_output=True, text=True
    )
    data = json.loads(result.stdout)
    
    # Subtitles are in data['subtitles'] or data['automatic_captions']
    return {
        'title': data.get('title'),
        'description': data.get('description'),
        'duration': data.get('duration'),
        'subtitles': data.get('automatic_captions', {})
    }

Search GitHub for Solutions

import subprocess
import json

def search_github_issues(query, repo=None):
    cmd = ["gh", "search", "issues", query, 
           "--limit", "20", "--json", 
           "title,url,state,body,comments"]
    if repo:
        cmd.extend(["--repo", repo])
    
    result = subprocess.run(cmd, capture_output=True, text=True)
    return json.loads(result.stdout)

# Search across all repos
issues = search_github_issues("memory leak in agents")

# Search specific repo
issues = search_github_issues("bug", repo="openai/gpt-4")

Monitor Reddit for Mentions

# Search and save results
rdt search "your_product_name" --limit 50 > mentions.txt

# Get specific subreddit
rdt subreddit r/artificial --limit 100

Read Web Page Content for Analysis

import requests

def get_clean_content(url):
    response = requests.get(
        f"https://r.jina.ai/{url}",
        headers={
            "X-With-Links-Summary": "true",
            "X-No-Cache": "true"
        }
    )
    return response.text

content = get_clean_content("https://news.ycombinator.com")

Troubleshooting

Doctor Command Shows ❌

Run diagnostics:

agent-reach doctor

Each ❌ includes a fix suggestion. Common issues:

Twitter/Reddit not working:

  • Need Cookie authentication
  • Use Cookie-Editor to export cookies
  • Run twitter configure or rdt login

Bilibili 403 on server:

  • Need proxy for non-CN IPs
  • Set HTTP_PROXY and HTTPS_PROXY env vars

GitHub rate limited:

  • Authenticate: gh auth login
  • Authenticated rate: 5,000/hour vs 60/hour

yt-dlp fails:

  • Update to latest: pip install -U yt-dlp
  • Tool is actively maintained, updates frequently

MCP Server Connection Issues

# Check mcporter status
mcporter list

# Restart a server
mcporter restart xiaohongshu

# View server logs
mcporter logs exa

Proxy Not Working

# Test proxy connection
curl -x http://proxy:port https://api.bilibili.com

# Set for specific command
export HTTPS_PROXY=http://proxy:port
yt-dlp URL

Cookie Expired

Re-export fresh cookies:

  1. Login to service in browser
  2. Export with Cookie-Editor
  3. Reconfigure CLI tool

Environment Variables

# Proxy (for server deployments)
export HTTP_PROXY=http://proxy:port
export HTTPS_PROXY=http://proxy:port

# GitHub
export GITHUB_TOKEN=ghp_xxxxx

# Exa (optional, for advanced features)
export EXA_API_KEY=your_key_here

# Custom config paths (optional)
export AGENT_REACH_CONFIG_DIR=~/.config/agent-reach

Safety & Privacy

  • All cookies stored locally in ~/.twitter-cli/, ~/.rdt-cli/, etc.
  • No data uploaded to Agent Reach servers (there are none)
  • Code is open source — audit anytime
  • Use --safe mode during install to review system package installs

Updating

# Update agent-reach
pip install -U agent-reach

# Update individual tools
pip install -U yt-dlp
gh extension upgrade --all
npm update -g mcporter

Check for breaking changes: https://github.com/Panniantong/agent-reach/blob/main/CHANGELOG.md

Platform Support Matrix

PlatformOut of BoxAfter ConfigNotes
Web✅—Jina Reader, no limits
YouTube✅—yt-dlp, 1800+ sites
RSS✅—feedparser
GitHub✅🔧 Auth for privategh CLI
Twitter🔧 Cookie🔧 Cookietwitter-cli
Reddit🔧 Cookie🔧 Cookierdt-cli
Bilibili✅ Local🔧 Proxy (server)yt-dlp
XiaoHongShu🔧 Cookie🔧 Cookiexhs-cli via MCP
Search🔧 MCP🔧 API key (optional)Exa
WeChat✅—Via Exa search
Weibo✅—Direct API
V2EX✅—Direct API

Legend: ✅ Works immediately | 🔧 Needs configuration

FAQ

What does the agent-reach-internet-access skill do?

Give AI agents eyes to see the internet — scrape Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu with zero API fees

How do I install agent-reach-internet-access?

Run npx skills add https://github.com/reason-machines/ai-agent-skills --skill agent-reach-internet-access in your project. The skills CLI detects the agents you use and copies the skill into the right folder. For Claude Code you can also place the skill folder at ~/.claude/skills/agent-reach-internet-access/ (personal) or .claude/skills/agent-reach-internet-access/ (project).

Which AI agents can use agent-reach-internet-access?

Any agent that supports the open Agent Skills format (a folder with a SKILL.md file), including Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf, Cline, OpenCode and more.

Is agent-reach-internet-access safe to install?

No partner security audit is available yet. Skills can run commands and scripts, so read the SKILL.md and any bundled files before installing.

Who maintains agent-reach-internet-access?

It is published by reason-machines in reason-machines/ai-agent-skills. It has 229 recorded installs.

More from reason-machines/ai-agent-skills

View all →

agency-agents-zh-skill

reason-machines/ai-agent-skills

Open the skill to read its SKILL.md instructions.

AI & Agents322 installs

agent-sprite-forge-2d-game-assets

reason-machines/ai-agent-skills

Generate 2D sprite sheets, animated GIFs, transparent PNGs, layered maps, and engine-ready prototypes for Godot/Unity using Codex-driven asset pipelines.

Media & Creative217 installs

agentmemory-persistent-memory

reason-machines/ai-agent-skills

Add persistent memory to AI coding agents using agentmemory - remembers context, preferences, and decisions across sessions

AI & Agents216 installs

awesome-agent-skills

reason-machines/ai-agent-skills

Open the skill to read its SKILL.md instructions.

AI & Agents214 installs

oh-my-openagent-orchestration

reason-machines/ai-agent-skills

Open the skill to read its SKILL.md instructions.

AI & Agents211 installs

tencentdb-agent-memory

reason-machines/ai-agent-skills

Local-first long-term memory for AI agents with symbolic short-term memory and layered long-term recall via a 4-tier progressive pipeline

AI & Agents204 installs

Popular ai & agents skills

View category →

google-agents-cli-publish

google/agents-cli

This skill should be used when the user wants to "publish an agent", "publish my ADK agent", "register an agent with Gemini", "publish to Gemini" (formerly Gemini Enterprise), or needs guidance on the agents-cli publish gemini command. Also use when the user wants to "manage agents in Agent Registry", "list/update/delete registered agents", or "register an MCP server". Covers ADK vs A2A registration modes, programmatic and interactive usage, flag reference, auto-detection from deployment metadata, Agent Registry fleet management, and troubleshooting. Part of the agents-cli skills suite. Do NOT use for deployment (use google-agents-cli-deploy).

AI & Agents513.9K installs

google-agents-cli-eval

google/agents-cli

This skill should be used when the user wants to "run an evaluation", "evaluate my agent", "evaluate my ADK agent", "write an eval dataset", "analyze eval failures", "compare eval results", "optimize agent", or needs guidance on the Gemini platform eval methodology and the Quality Flywheel. Covers eval metrics, dataset schema, LLM-as-judge scoring, and common failure causes. Applies to any agents-cli project, whatever framework the agent is written in. Do NOT use for agent API code patterns (ADK: use google-agents-cli-adk-code), deployment (use google-agents-cli-deploy), or project scaffolding (use google-agents-cli-scaffold).

AI & Agents513.5K installs

skill-creator

anthropics/skills

Create new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.

OfficialAI & Agents416.4K installs

codex-pet

prime-skills/runcomfy-agent-skills

Codex Pet generator on RunComfy. Build a Codex-compatible Codex Pet spritesheet.webp + pet.json from a single reference image, drop it into ${CODEXHOME:-$HOME/.codex}/pets/ / and Codex picks it up as a custom Codex Pet next to the 8 built-ins. This skill produces the exact Codex Pet atlas Codex expects (1536x1872 PNG/WebP, 8 cols x 9 rows, 192x208 cells, 9 animation states — idle, running-right, running-left, waving, jumping, failed, waiting, running, review). Calls OpenAI GPT Image 2 edit ONCE via the local RunComfy CLI as runcomfy run openai/gpt-image-2/edit to produce a canonical Codex Pet pose, then assembles all 9 animation rows programmatically with ImageMagick micro-transforms — no Codex Pro, no $imagegen, no OPENAIAPIKEY required, only RUNCOMFYTOKEN. Triggers on "codex pet", "create codex pet", "make codex pet", "hatch codex pet", "/hatch image", "desktop pet codex", "codex pets", "spritesheet.webp", or any explicit ask to build a custom pet for OpenAI Codex.

AI & Agents402.1K installs

full-output-enforcement

leonxlnx/taste-skill

Overrides default LLM truncation behavior. Enforces complete code generation, bans placeholder patterns, and handles token-limit splits cleanly. Apply to any task requiring exhaustive, unabridged output.

AI & Agents401K installs

executing-plans

obra/superpowers

Use when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available

AI & Agents245.7K installs