MCPInstall
Web Crawler logo

Web Crawler

by orange-fruit01 orange-fruit01/mcp-test-run

Enables web crawling capabilities for extracting website content in markdown format, containerized with Docker for easy deployment on Render.com and accessible through a health check endpoint.

READMEmain

MCP Service

This is a Model Control Protocol (MCP) service that provides tools for Cursor and Claude applications.

Connecting to the Service

For Cursor Users

  1. Open Cursor
  2. Go to Settings > AI > Advanced
  3. Under "MCP Endpoint", enter: https://your-render-url.onrender.com
  4. Save settings and restart Cursor

For Claude App Users

  1. Open Claude App
  2. Go to Settings > Advanced
  3. Under "MCP Connection", enter: https://your-render-url.onrender.com
  4. Save settings

Available Tools

  • Web Crawler: Crawls web pages and returns content as markdown

Deployment

This service is deployed on Render.com as a Docker container. The service is accessible at:

https://your-render-url.onrender.com

Replace your-render-url.onrender.com with the actual URL provided by Render after deployment.

Environment Variables

  • MCP_HOST: Host to bind the server (default: 0.0.0.0)
  • MCP_PORT: Port to bind the server (default: 8000)

Monitoring

A health endpoint is available at /health for monitoring the service status.

More Web Scraping & Data Collection MCP servers

Similar servers from the same category.

BuiltWith

builtwith

Integrates with BuiltWith's technology detection API to identify and analyze web technologies used on specified domains, providing marketers and developers with comprehensive website stack information.

Web Scraping & Data Collection

Provides a bridge to ElfProxy's dynamic IP proxy service for retrieving country-specific proxy IPs, enabling IP rotation, geo-targeting, and bypassing regional restrictions.

WebOptimize

UniFuncs

unifuncs

Provides a bridge to the UniFuncs API for web search and web reading capabilities through TypeScript implementation with Express and NPX commands.

search

Integrates with Bocha, a Chinese search engine, to provide web search capabilities with rich structured data from billions of documents across multiple domains and languages.

search