cosmic-mcp

All servers

robotstxt-ai

io.github.sharozdawa/robotstxt-ai · version 1.0.1

Community listing. Not installed or verified by cosmic-mcp. Read the source and its permissions before you run it.

Analyze and generate robots.txt with AI crawler detection for 20+ bots like GPTBot and ClaudeBot.

https://github.com/sharozdawa/robotstxt-ai

Quick facts

  • Listing: Community
  • Published Mar 21, 11:04 UTC
  • Transport: stdio
  • Package: npm robotstxt-ai-mcp 1.0.1
  • Listed by cosmic-mcp: Nov 9, 04:27 UTC

Web scraping and crawling

Setup snippets

Generated from the registry entry. cosmic-mcp has not run or reviewed this server; values in angle brackets are yours to fill in.

Run it locally (npm)

npx -y '[email protected]'

Claude Desktop (claude_desktop_config.json)

{
  "mcpServers": {
    "robotstxt-ai": {
      "args": [
        "-y",
        "[email protected]"
      ],
      "command": "npx"
    }
  }
}

Claude Code

claude mcp add 'robotstxt-ai' -- npx -y '[email protected]'

Cursor (.cursor/mcp.json)

{
  "mcpServers": {
    "robotstxt-ai": {
      "args": [
        "-y",
        "[email protected]"
      ],
      "command": "npx"
    }
  }
}

VS Code (.vscode/mcp.json)

{
  "servers": {
    "robotstxt-ai": {
      "args": [
        "-y",
        "[email protected]"
      ],
      "command": "npx",
      "type": "stdio"
    }
  }
}

Related servers

  • 1Claw Vault
    HSM-backed vault secrets for AI agents (JIT fetch) plus prompt-injection and threat scanning.
  • 222wcnm-bilistalkermcp Verified domain
    Track Bilibili creators and get the latest updates on videos, dynamics, and articles. Fetch user p…
  • 3d-texture-toolkit
    Extract embedded PBR texture maps from local GLB files.
  • 402registry Verified domain
    Does an AI crawler see your site? Free registry lookup, x402 pricing; checks paid in USDC on Base.
  • 4fetch
    Fetch a URL and get clean Markdown with metadata. No API key required; rate-limited per IP.
  • AEO Auditor — AI Search & Agent Readiness Engine Verified domain
    Live AEO, GEO, JSON-LD Schema, robots.txt AI crawler, and Agent Readiness diagnostic auditor.