cosmic-mcp

All servers

Agent Crawl Readiness

io.github.sadri-dridi/agent-crawl-readiness · version 1.0.0

Community listing. Not installed or verified by cosmic-mcp. Read the source and its permissions before you run it.

Bounded crawl-readiness check for agent access: robots.txt, sitemap.xml, llms.txt, homepage.

https://github.com/sadri-dridi/named-mcp-utilities

Quick facts

  • Listing: Community
  • Published Sep 23, 20:55 UTC
  • Transport: streamable-http
  • Listed by cosmic-mcp: Nov 9, 04:27 UTC

Web scraping and crawling

Setup snippets

Generated from the registry entry. cosmic-mcp has not run or reviewed this server; values in angle brackets are yours to fill in.

Remote MCP URL (Claude and ChatGPT custom connectors)

https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcp

Claude Code

claude mcp add --transport http 'agent-crawl-readiness' 'https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcp'

Cursor (.cursor/mcp.json)

{
  "mcpServers": {
    "agent-crawl-readiness": {
      "url": "https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcp"
    }
  }
}

VS Code (.vscode/mcp.json)

{
  "servers": {
    "agent-crawl-readiness": {
      "type": "http",
      "url": "https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcp"
    }
  }
}

Related servers

  • 1Claw Vault
    HSM-backed vault secrets for AI agents (JIT fetch) plus prompt-injection and threat scanning.
  • 222wcnm-bilistalkermcp Verified domain
    Track Bilibili creators and get the latest updates on videos, dynamics, and articles. Fetch user p…
  • 3d-texture-toolkit
    Extract embedded PBR texture maps from local GLB files.
  • 402registry Verified domain
    Does an AI crawler see your site? Free registry lookup, x402 pricing; checks paid in USDC on Base.
  • 4fetch
    Fetch a URL and get clean Markdown with metadata. No API key required; rate-limited per IP.
  • AEO Auditor — AI Search & Agent Readiness Engine Verified domain
    Live AEO, GEO, JSON-LD Schema, robots.txt AI crawler, and Agent Readiness diagnostic auditor.