cosmic-mcp

All servers

polite-crawler

io.github.mark1russell7/polite-crawler · version 0.5.2

Community listing. Not installed or verified by cosmic-mcp. Read the source and its permissions before you run it.

A polite crawler for agents: obeys robots.txt, fetches each page once, finds and verifies quotes.

https://github.com/mark1russell7/crawler

Quick facts

  • Listing: Community
  • Published Nov 9, 00:59 UTC
  • Transport: stdio
  • Package: npm polite-crawler-mcp 0.5.2
  • Listed by cosmic-mcp: Nov 9, 04:26 UTC

Web scraping and crawling

Setup snippets

Generated from the registry entry. cosmic-mcp has not run or reviewed this server; values in angle brackets are yours to fill in.

Run it locally (npm)

npx -y '[email protected]'

Claude Desktop (claude_desktop_config.json)

{
  "mcpServers": {
    "polite-crawler": {
      "args": [
        "-y",
        "[email protected]"
      ],
      "command": "npx",
      "env": {
        "POLITE_CRAWLER_ALLOW_PRIVATE": "<your polite crawler allow private>",
        "POLITE_CRAWLER_ALLOW_USER_PROVIDED": "<your polite crawler allow user provided>",
        "POLITE_CRAWLER_CONTACT": "<your polite crawler contact>",
        "POLITE_CRAWLER_DIR": "<your polite crawler dir>",
        "POLITE_CRAWLER_SIGNATURE_AGENT": "<your polite crawler signature agent>",
        "POLITE_CRAWLER_SIGN_KEY": "<your polite crawler sign key>"
      }
    }
  }
}

Claude Code

claude mcp add 'polite-crawler' -- npx -y '[email protected]'

Cursor (.cursor/mcp.json)

{
  "mcpServers": {
    "polite-crawler": {
      "args": [
        "-y",
        "[email protected]"
      ],
      "command": "npx",
      "env": {
        "POLITE_CRAWLER_ALLOW_PRIVATE": "<your polite crawler allow private>",
        "POLITE_CRAWLER_ALLOW_USER_PROVIDED": "<your polite crawler allow user provided>",
        "POLITE_CRAWLER_CONTACT": "<your polite crawler contact>",
        "POLITE_CRAWLER_DIR": "<your polite crawler dir>",
        "POLITE_CRAWLER_SIGNATURE_AGENT": "<your polite crawler signature agent>",
        "POLITE_CRAWLER_SIGN_KEY": "<your polite crawler sign key>"
      }
    }
  }
}

VS Code (.vscode/mcp.json)

{
  "servers": {
    "polite-crawler": {
      "args": [
        "-y",
        "[email protected]"
      ],
      "command": "npx",
      "env": {
        "POLITE_CRAWLER_ALLOW_PRIVATE": "<your polite crawler allow private>",
        "POLITE_CRAWLER_ALLOW_USER_PROVIDED": "<your polite crawler allow user provided>",
        "POLITE_CRAWLER_CONTACT": "<your polite crawler contact>",
        "POLITE_CRAWLER_DIR": "<your polite crawler dir>",
        "POLITE_CRAWLER_SIGNATURE_AGENT": "<your polite crawler signature agent>",
        "POLITE_CRAWLER_SIGN_KEY": "<your polite crawler sign key>"
      },
      "type": "stdio"
    }
  }
}

Related servers

  • 1Claw Vault
    HSM-backed vault secrets for AI agents (JIT fetch) plus prompt-injection and threat scanning.
  • 222wcnm-bilistalkermcp Verified domain
    Track Bilibili creators and get the latest updates on videos, dynamics, and articles. Fetch user p…
  • 3d-texture-toolkit
    Extract embedded PBR texture maps from local GLB files.
  • 402registry Verified domain
    Does an AI crawler see your site? Free registry lookup, x402 pricing; checks paid in USDC on Base.
  • 4fetch
    Fetch a URL and get clean Markdown with metadata. No API key required; rate-limited per IP.
  • AEO Auditor — AI Search & Agent Readiness Engine Verified domain
    Live AEO, GEO, JSON-LD Schema, robots.txt AI crawler, and Agent Readiness diagnostic auditor.