cosmic-mcp

All servers

LLM Eval Harness

io.github.razilzul/llm-eval-harness ยท version 1.0.0

Community listing. Not installed or verified by cosmic-mcp. Read the source and its permissions before you run it.

Run a golden set and gate LLM model swaps on accuracy, with latency and cost per request.

https://github.com/razilzul/llm-eval-harness

Quick facts

  • Listing: Community
  • Published Oct 18, 13:22 UTC
  • Transport: stdio
  • Package: oci ghcr.io/razilzul/llm-eval-harness:1.0.0
  • Listed by cosmic-mcp: Nov 9, 04:26 UTC

AI and LLMs

Setup snippets

Generated from the registry entry. cosmic-mcp has not run or reviewed this server; values in angle brackets are yours to fill in.

Run it locally (oci)

docker run -i --rm -e 'ANTHROPIC_API_KEY' -e 'GEMINI_API_KEY' 'ghcr.io/razilzul/llm-eval-harness:1.0.0'

Claude Desktop (claude_desktop_config.json)

{
  "mcpServers": {
    "llm-eval-harness": {
      "args": [
        "run",
        "-i",
        "--rm",
        "-e",
        "ANTHROPIC_API_KEY",
        "-e",
        "GEMINI_API_KEY",
        "ghcr.io/razilzul/llm-eval-harness:1.0.0"
      ],
      "command": "docker",
      "env": {
        "ANTHROPIC_API_KEY": "<your anthropic api key>",
        "GEMINI_API_KEY": "<your gemini api key>"
      }
    }
  }
}

Claude Code

claude mcp add 'llm-eval-harness' -- docker run -i --rm -e 'ANTHROPIC_API_KEY' -e 'GEMINI_API_KEY' 'ghcr.io/razilzul/llm-eval-harness:1.0.0'

Cursor (.cursor/mcp.json)

{
  "mcpServers": {
    "llm-eval-harness": {
      "args": [
        "run",
        "-i",
        "--rm",
        "-e",
        "ANTHROPIC_API_KEY",
        "-e",
        "GEMINI_API_KEY",
        "ghcr.io/razilzul/llm-eval-harness:1.0.0"
      ],
      "command": "docker",
      "env": {
        "ANTHROPIC_API_KEY": "<your anthropic api key>",
        "GEMINI_API_KEY": "<your gemini api key>"
      }
    }
  }
}

VS Code (.vscode/mcp.json)

{
  "servers": {
    "llm-eval-harness": {
      "args": [
        "run",
        "-i",
        "--rm",
        "-e",
        "ANTHROPIC_API_KEY",
        "-e",
        "GEMINI_API_KEY",
        "ghcr.io/razilzul/llm-eval-harness:1.0.0"
      ],
      "command": "docker",
      "env": {
        "ANTHROPIC_API_KEY": "<your anthropic api key>",
        "GEMINI_API_KEY": "<your gemini api key>"
      },
      "type": "stdio"
    }
  }
}

Related servers

  • Harness Verified domain
    Start Claude Code and Codex agents the person can watch, wait for them and read their answers.
  • docs-mcp Verified domain
    Remote MCP server for Tandem docs, install guides, SDKs, workflows, and agent setup help.
  • spotdb Verified domain
    Ephemeral data sandbox for AI workflows with guardrails and security
  • Adako: ad ops for AI assistants Verified domain
    Google, Meta, LinkedIn and ChatGPT Ads for AI assistants. Every change needs your approval.
  • Aidoo for Odoo Verified domain
    Query and update your Odoo ERP in plain language, under your own Odoo access rules.
  • EXTREMELY agency Verified domain
    Your agency of AI agents in your AI app: what agents did, approvals, requests, jobs and reports.