💡 claude plugin eval: A Reproducible, CI-Ready Evaluation Runner for Your Plugins

Claude Code v2.1.269 ships the most significant plugin development feature since the plugin system launched: claude plugin eval, a built-in evaluation runner that executes a suite of test prompts against your plugin, scores outputs using one of six grader types, and produces a JSON + HTML report you can gate in CI. Crucially, it also runs the same prompts against a no-plugin baseline — so you get an empirical answer to the question "does my plugin actually help?"

The six grader types

Four graders are free (no additional API calls); two call a judge model and consume tokens:

• exact — string equality after normalisation. Use for deterministic outputs (IDs, structured JSON keys).
• contains — checks that the response includes a required substring. Use for mandatory disclaimers, required field mentions, or keyword presence.
• regex — matches against a regular expression. Use for format validation (phone numbers, dates, codes).
• json-schema — validates response against a JSON Schema. Use for any plugin that returns structured data.
• llm-judge — sends the prompt, expected criteria, and Claude's response to a judge model for a pass/fail verdict with reasoning. Use for open-ended quality or tone requirements.
• rubric — a scored variant of llm-judge that returns a 0–10 score on each rubric dimension. Use when you want aggregate quality tracking over time.

Running an eval

# Run eval suite defined in plugin-eval.yaml
claude plugin eval --config plugin-eval.yaml

# Output: JSON report + HTML visualisation in ./eval-results/
# Exit code 0 if all pass-threshold checks pass, 1 if any fail
# --baseline flag included by default; suppress with --no-baseline

The eval config file declares test cases, expected outputs, and which grader(s) to use per case. A minimal two-grader example:

version: 1
plugin: my-plugin
threshold: 0.85   # overall pass rate required; CI fails below this
cases:
  - id: basic-lookup
    prompt: "What is the capital of France?"
    graders:
      - type: contains
        value: "Paris"
  - id: tone-check
    prompt: "Explain quantum entanglement in one sentence."
    graders:
      - type: llm-judge
        criteria: "Response is accurate, uses an analogy, and is under 30 words."

- name: Plugin eval gate
  run: claude plugin eval --config plugin-eval.yaml
  env:
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

The baseline comparison

This is the feature most plugin developers have been waiting for. Without a baseline, a plugin's 82% pass rate on your eval suite tells you nothing — is that good? Bad? The baseline run executes the same prompts with the plugin disabled and scores them with the same graders. If Claude scores 79% without your plugin, your plugin is adding 3 percentage points of net value. If Claude scores 84% without it, your plugin is actively hurting quality and you need to investigate why before shipping.

💡 Four More v2.1.269 Upgrades Worth Knowing: Output Styles, Bash Diffs, Agent Maps, and Concurrent Limits

Alongside the plugin eval runner, v2.1.269 ships four quality-of-life improvements that each address a common friction point in Claude Code workflows. None of these require configuration changes to start using — they are all available immediately after upgrading.

/output-style [name] — switch response format mid-session

The new /output-style slash command lets you change Claude Code's output format inside a running remote or headless session without restarting. Available styles include compact (terse, no explanatory prose — useful for CI log parsing), verbose (full reasoning traces — useful for debugging), and markdown (default for interactive sessions). This is especially useful in remote sessions where you cannot interactively re-launch with different flags.

# Switch to compact output for the rest of this remote session
/output-style compact

# Back to markdown
/output-style markdown

Bash tool diff panel — see exactly what changed

When Claude Code's Bash tool writes or modifies a file, a new diff panel now appears inline showing the precise character-level changes made — the same format as git diff. This was one of the top-requested improvements from Claude Code users: previously, you had to run git diff yourself to confirm whether a Bash command had the intended effect. The diff panel appears automatically for any Bash invocation that modifies tracked files; it can be dismissed with Escape or suppressed globally with --no-bash-diff.

VSCode agent map — visual sub-agent orchestration

The VSCode extension gains an agent map: a sidebar panel that renders your active Claude Code session as a graph, showing the root agent and all spawned sub-agents as nodes with per-agent status cards (running, waiting, completed, failed). Edges represent the spawn relationship. This makes it much easier to understand what is happening in complex multi-agent workflows — particularly when a sub-agent has stalled or is consuming unexpectedly high resources. Clicking a node opens that agent's tool-call log.

Concurrent agent limits via CLAUDE_CODE_WORKFLOW_MAX_CONCURRENT_AGENTS

A new environment variable caps how many sub-agents can run simultaneously within a single Claude Code session. The default remains uncapped for backward compatibility, but setting this variable prevents resource exhaustion on shared machines or in environments with API rate limits:

# Allow at most 4 concurrent sub-agents (others queue until a slot opens)
export CLAUDE_CODE_WORKFLOW_MAX_CONCURRENT_AGENTS=4

When the cap is hit, additional Task tool invocations queue internally and resume as slots free up. This pairs well with the new agent map — you can watch the queue drain in real time from the VSCode sidebar.