MCPInstall
mcp-workflows-marketplace logo

mcp-workflows-marketplace

by truaxki truaxki/mcp-workflows-marketplace
View on GitHub
READMEmain

MCP Servers Workflow Marketplace

This repository contains tools for scraping, analyzing, and managing Model Context Protocol (MCP) servers and their capabilities to create an intelligent recommendation system for agentic workflows.

Vision

The MCP Servers Workflow Marketplace aims to revolutionize how users interact with MCP servers by creating an intelligent recommendation system that:

  1. Understands User Intent: Intelligently reasons about the task a user wants to accomplish
  2. Recommends Optimal Tools: Suggests the most appropriate MCP servers and tools based on the task requirements
  3. Constructs Agentic Workflows: Automatically builds and orchestrates workflows that combine multiple MCP servers and tools
  4. Optimizes for Efficiency: Leverages a comprehensive knowledge base of over 2,000 MCP servers to find the best solutions

By combining relational databases (SQLite), vector databases (ChromaDB), and graph databases (Neo4j), this project creates a powerful knowledge system that understands the relationships between tasks, tools, and workflows.

Architecture

The system processes and analyzes MCP servers through multiple stages:

  1. Data Collection: Pull all PulseMCP API data to a relational database (SQLite)
  2. Data Enhancement: Enrich the database by adding integrations, tools, prompts, and resources
  3. Vector Embeddings: Process README files and documentation into vector embeddings for semantic search
  4. Graph Relationships: Map relationships between servers, tools, tasks, and workflows in Neo4j
  5. Recommendation Engine: Use the combined data sources to intelligently recommend and construct workflows

Key Relationships

  • Servers contain tools
  • Tasks require tools
  • Tools can belong to a task stack
  • Workflows are made up of multiple tools

Features

  • Enhanced MCP Scraper: Scrapes GitHub repositories to extract MCP server tools, resources, features, and integrations
  • Multiple extraction strategies: Uses GitHub API, web scraping, and Firecrawl (if available) with weighted consensus
  • Async HTTP requests: Improves performance with asynchronous requests
  • Database versioning: Tracks changes to tool definitions over time
  • Tool metadata enrichment: Categorizes tools based on their functionality
  • Vector search: Enables semantic search across server documentation
  • Workflow construction: Intelligently combines tools into effective workflows
  • Task analysis: Understands and decomposes user tasks into required capabilities

Featured MCP Servers

This project is designed to amplify the usage of powerful MCP servers that provide a broad range of tools and capabilities. A notable example is:

Toolhouse MCP Server

The Toolhouse MCP Server provides an extensive collection of tools and capabilities that can be leveraged in various workflows. Our marketplace is specifically designed to help users discover and utilize such powerful servers effectively. The Toolhouse server is a perfect example of the kind of comprehensive tool suite this marketplace aims to promote. Example use cases include:

  • Combining Toolhouse's data processing capabilities with specialized domain servers
  • Creating multi-step workflows that leverage Toolhouse's versatile toolset
  • Recommending Toolhouse as a comprehensive solution for complex tasks

Installation

This project uses uv for package management.

# Install uv if you don't have it
curl -LsSf https://astral.sh/uv/install.sh | sh  # Unix/macOS
# or
powershell -c "irm https://astral.sh/uv/install.ps1 | iex"  # Windows

# Clone the repository
git clone https://github.com/yourusername/mcp-servers-workflow-marketplace.git
cd mcp-servers-workflow-marketplace

# Install the package and its dependencies
uv pip install -e .

# Install with optional GitHub API support
uv pip install -e ".[github]"

# Install with development tools (black, isort, mypy, pytest)
uv pip install -e ".[dev]"

For development, you can use uv's sync command to keep dependencies in sync with pyproject.toml:

# Sync dependencies with pyproject.toml
uv pip sync

Usage

MCP Server Inventory

Pull all servers from the PulseMCP API and store them in a SQLite database:

python mcp-server-inventory.py

README Downloader

Download README files from GitHub repositories of MCP servers:

# Download README files for all servers
python mcp-readme-downloader.py

# Limit to a specific number of servers
python mcp-readme-downloader.py --limit 10

# Force download even if README exists
python mcp-readme-downloader.py --force

# Enable concurrent downloads
python mcp-readme-downloader.py --concurrent

Enhanced MCP Scraper

Extract tools from MCP server repositories using different methods:

# Scrape using all available methods
python enhanced-mcp-scraper.py

# Limit to a specific number of servers
python enhanced-mcp-scraper.py --limit 10

# Use a specific extraction method (all, github, web, firecrawl)
python enhanced-mcp-scraper.py --method web

# Run extraction methods concurrently
python enhanced-mcp-scraper.py --concurrent

Embedding README Files

Process README files into vector embeddings for semantic search:

# Embed all README files
python embed-readme.py

# Limit to a specific number of README files
python embed-readme.py --limit 10

# Force embedding even if already processed
python embed-readme.py --force

# Set batch size for embeddings
python embed-readme.py --batch_size 64

Workflow Management

Future functionality will include:

  • Refreshing server marketplace data
  • Refreshing installed servers
  • Configuring tasks and settings
    • Describing tasks (input/output data types, LLM locality/provider)
  • Auto-installing servers based on task requirements

Environment Variables

Create a .env file with the following variables (see .env.example for a complete list):

# Database Configuration
DB_PATH=./data/pulsemcp-inventory.db

# API Configuration
PULSEMCP_API_URL=https://api.pulsemcp.com/v0beta/servers
PULSEMCP_REQUEST_DELAY=0.5

# Scraper Configuration
SCRAPER_LOG_DIR=./log
SCRAPER_USER_AGENT=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36
SCRAPER_REQUEST_DELAY=1.0
SCRAPER_MAX_WORKERS=5

# GitHub API Configuration
GITHUB_TOKEN=your_github_token_here

# Embedder Configuration
EMBEDDER_README_DIR=./data/README_FILES
EMBEDDER_CHROMA_DIR=./data/chroma_db
OLLAMA_BASE_URL=http://localhost:11434
EMBEDDING_MODEL=nomic-embed-text

Project Structure

  • mcp-server-inventory.py: Script to fetch all servers from PulseMCP API
  • mcp-readme-downloader.py: Script to download README files from GitHub repositories
  • enhanced-mcp-scraper.py: Script for scraping MCP server tools
  • embed-readme.py: Script for embedding README files into vector database
  • mcp_servers_workflow_marketplace/: Package directory containing the modular code
  • data/: Directory for storing databases and other data
    • data/pulsemcp-inventory.db: SQLite database for server inventory
    • data/README_FILES/: Directory for downloaded README files
    • data/chroma_db/: Directory for ChromaDB vector database
  • log/: Directory for log files

Development

Running Tests

Run the tests using uv and pytest:

uv run pytest -v

Code Style

Format your code with Black and isort:

uv run black .
uv run isort .

Type Checking

Run type checking with mypy:

uv run mypy mcp_servers_workflow_marketplace

Dependencies

  • aiohttp: For asynchronous HTTP requests
  • beautifulsoup4: For parsing HTML
  • python-dotenv: For loading environment variables
  • PyGithub (optional): For GitHub API access
  • firecrawl (optional): For enhanced extraction capabilities
  • chromadb: For vector database storage
  • neo4j: For graph database relationships