Skip to content

Add a Copilot skill eval pilot - #6484

Open
aaronpowell wants to merge 7 commits into
mainfrom
aaronpowell-skill-eval-pilot
Open

aaronpowell wants to merge 7 commits into
mainfrom
aaronpowell-skill-eval-pilot

Conversation

@aaronpowell

@aaronpowell aaronpowell commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Pull Request Checklist

  • I have read and followed the CONTRIBUTING.md guidelines.
  • I have read and followed the Guidance for submissions involving paid services.
  • My contribution adds a new instruction, prompt, agent, skill, workflow, or canvas extension file in the correct directory.
  • The file follows the required naming convention.
  • The content is clearly structured and follows the example format.
  • I have tested my instructions, prompt, agent, skill, workflow, or canvas extension with GitHub Copilot.
  • I have run npm start and verified that README.md is up to date.
  • I am targeting the main branch for this pull request.

Description

Pilot behavioral evaluation for convert-plaintext-to-md so skill changes can be assessed for behavior as well as structure. The read-only Vally workflow lints affected eval specifications alongside skills, including on fork PRs. A separate advisory workflow runs the behavioral eval on trusted changes and publishes a job summary and report artifact; the existing Submission Gate comment will report its status once the gate configuration is on main. The eval pipeline preserves Vally's exit status while logging its output.


Type of Contribution

  • New instruction file.
  • New prompt file.
  • New agent file.
  • New plugin.
  • New skill file.
  • New agentic workflow.
  • New canvas extension.
  • Update to existing instruction, prompt, agent, plugin, skill, workflow, or canvas extension.
  • Other (please specify): GitHub Actions workflow and skill eval specification.

Additional Notes

  • node --test .github/scripts/review/submission-gate.test.mjs: passed (50 tests), including a regression test for successful and failing Vally commands.
  • npx --yes @microsoft/vally-cli@0.21.0 lint --eval-spec evals/skills/convert-plaintext-to-md/eval.yaml --strict: passed.
  • npm start: passed; generated READMEs were up to date.
  • bash .github/scripts/maintenance/fix-line-endings.sh: passed.
  • go run github.com/rhysd/actionlint/cmd/actionlint@latest .github/workflows/skill-check.yml .github/workflows/skill-check-comment.yml .github/workflows/skill-eval.yml: failed because actionlint does not recognize the copilot-requests permission.
  • go run github.com/rhysd/actionlint/cmd/actionlint@latest -ignore 'unknown permission scope "copilot-requests"' .github/workflows/skill-eval.yml: passed.

The latest live eval run correctly reports failure: six of seven graders passed, but the shell-code-block grader did not. The summary and full report artifact were still published. An earlier run falsely reported success because tee masked Vally's exit code; the workflow now uses pipefail to prevent this. Behavioral evals require Copilot CLI billing and do not run on fork PRs. The grader regexes compile with Vally 0.21.0, which supports the inline flags used by the pilot.


By submitting this pull request, I confirm that my contribution abides by the Code of Conduct and will be licensed under the MIT License.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@aaronpowell
aaronpowell requested a review from a team as a code owner November 18, 2026 01:13
Copilot AI adaptive review requested due to automatic review settings November 18, 2026 01:14
@github-actions github-actions Bot added new-submission PR adds at least one new contribution workflow PR touches workflow automation awaiting-automation merge-risk:high labels Nov 18, 2026
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

🚦 Submission status: 👀 Ready for review

Risk tier: merge-risk:high — Privileged execution, automation, or review-policy change
Required to merge: passing submission-gate checks plus 1 approval from reviewers with write access, including a maintainer with admin or maintain permission.

Why this tier
  • .github/scripts/review/submission-gate.test.mjs is a high-risk path (automation, scripts, MCP config, hooks, or review policy)
  • .github/submission-gate.yml is a high-risk path (automation, scripts, MCP config, hooks, or review policy)
  • .github/workflows/skill-check-comment.yml is a high-risk path (automation, scripts, MCP config, hooks, or review policy)
  • .github/workflows/skill-check.yml is a high-risk path (automation, scripts, MCP config, hooks, or review policy)
  • .github/workflows/skill-eval.yml is a high-risk path (automation, scripts, MCP config, hooks, or review policy)

Automated checks

Check Status Details
Line endings ✅ Passed Passed · logs
Spelling ✅ Passed Passed · logs
Generated README consistency ✅ Passed Passed · logs
Skill lint (vally) ✅ Passed Passed · logs
Submission gate tests ✅ Passed Passed · logs
Contributor reputation ✅ Passed Passed · logs
Duplicate resource scan ✅ Passed Passed · logs
PR quality signal ✅ Passed Passed · logs

Review

  • Approvals: 0/1
  • Assigned reviewer: not assigned yet — comment /request-review to ask for one
  • Review target date: not set
  • Still needed: 1 more approval(s); an approval from a maintainer with admin or maintain permission
  • The core-maintainers pool is not staffed yet; an approver with admin or maintain permission is required instead.

Commands

Command Who What it does
/rerun-checks PR author, maintainers Re-runs failed or incomplete checks and re-evaluates this gate
/request-review PR author, maintainers Asks the review rotation to assign a reviewer (adds needs-reviewer)

Updated for 54ff8ab · gate run · This comment is maintained automatically — see submission gate docs.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

🟡 Contributor Reputation Check: MEDIUM risk

Check Risk
Profile MEDIUM
Credential audit NONE

Maintainers: please review this contributor before merging.
See the workflow run for full details.
Automated check powered by AGT.

@github-actions github-actions Bot added the needs-review:MEDIUM Contributor reputation check flagged MEDIUM risk label Nov 18, 2026
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

🔍 Vally Lint Results

✅ All checks passed

Scope Checked
Skills 0
Agents 0
Total 0
Severity Count
❌ Errors 0
⚠️ Warnings 0
ℹ️ Advisories 0

Summary

Level Finding
ℹ️ ✔ evals/skills/convert-plaintext-to-md/eval.yaml is valid
ℹ️ ✔ evals/skills/create-architectural-decision-record/eval.yaml is valid
ℹ️ ✔ evals/skills/editorconfig/eval.yaml is valid
ℹ️ ✔ evals/skills/update-markdown-file-index/eval.yaml is valid
Full linter output
### Linting eval spec evals/skills/convert-plaintext-to-md/eval.yaml
✔ evals/skills/convert-plaintext-to-md/eval.yaml is valid

### Linting eval spec evals/skills/create-architectural-decision-record/eval.yaml
✔ evals/skills/create-architectural-decision-record/eval.yaml is valid

### Linting eval spec evals/skills/editorconfig/eval.yaml
✔ evals/skills/editorconfig/eval.yaml is valid

### Linting eval spec evals/skills/update-markdown-file-index/eval.yaml
✔ evals/skills/update-markdown-file-index/eval.yaml is valid

@github-actions github-actions Bot added ready-for-review Submission passed intake validation and is ready for maintainer review and removed awaiting-automation labels Nov 18, 2026

stimuli:
- name: convert-deployment-notes
prompt: |
Comment thread .github/workflows/skill-eval.yml Outdated

permissions:
contents: read
copilot-requests: write

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The workflow can mask evaluation failures, while the graders can reject valid formatting, miss an omitted approval prerequisite, and conceal regressions in skill-defined output rules.

4 open issues
What changed in this PR

Adds a behavioral evaluation pilot for convert-plaintext-to-md, complementing structural skill validation with advisory results in the Submission Gate.

Changes:

  • Adds a deployment-notes fixture and graders for Markdown formatting, content preservation, and source-file preservation.
  • Extends read-only Vally linting and result counts to eval specifications, including fork PRs.
  • Adds a trusted-change evaluation workflow with job summaries, report artifacts, and advisory gate integration.
File Description
evals/​skills/​convert-plaintext-to-md/​fixtures/​deployment.txt Adds sample deployment notes for conversion.
evals/​skills/​convert-plaintext-to-md/​eval.yaml Defines the conversion scenario, graders, and passing threshold.
docs/​maintainers/​submission-gate.md Documents eval-spec linting and advisory behavioral checks.
.github/​workflows/​skill-eval.yml Runs behavioral evaluations and publishes reports for trusted changes.
.github/​workflows/​skill-check.yml Detects and lints affected eval specifications.
.github/​workflows/​skill-check-comment.yml Includes eval-spec counts in lint comments.
.github/​submission-gate.yml Registers eval paths and the advisory behavioral check.

Review effort: Max (adaptive)

Comment thread .github/workflows/skill-eval.yml Resolved
Comment on lines +21 to +25
Use the convert-plaintext-to-md skill to convert deployment.txt to GitHub-flavored
Markdown. Create deployment.txt.md in the same directory and leave deployment.txt
unchanged. Use a level-one heading for the title, preserve the two numbered steps,
put the shell command in a fenced shell code block, and preserve all technical
details and wording.
- type: file-matches
config:
path: deployment.txt.md
pattern: '(?s)```(?:bash|sh)\s*\r?\n\s*systemctl restart demo-api\.service\s*\r?\n```'
- type: file-matches
config:
path: deployment.txt.md
pattern: 'health endpoint returns HTTP 200'
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI adaptive review requested due to automatic review settings November 18, 2026 01:58
@github-actions github-actions Bot added awaiting-automation and removed ready-for-review Submission passed intake validation and is ready for maintainer review labels Nov 18, 2026
@github-actions github-actions Bot added ready-for-review Submission passed intake validation and is ready for maintainer review and removed awaiting-automation labels Nov 18, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The eval graders can reject valid fenced code blocks yet award full credit to conversions that omit the maintenance-window approval prerequisite.

3 open issues
2 previously missed issues

In code that hasn't changed since last review

Medium severity Pin eval-spec lint to the runner's Vally version

.github/​workflows/​skill-check.yml:164

Eval-spec lint resolves the latest Vally release, while .github/workflows/skill-eval.yml:60 pins execution to 0.21.0. As new releases ship, the two checks can disagree about supported schema features or regex syntax for the same spec. Pin this lint command to the runner's version so validation stays consistent.

Medium severity Require verified skill activation in the eval

evals/​skills/​convert-plaintext-to-md/​eval.yaml:26

All graders inspect files or the source diff, so a conversion produced without activating convert-plaintext-to-md can still receive full credit. Since the prompt already supplies the graded formatting rules, the pilot can miss regressions in skill activation. Require verified activation of this skill as a scored condition, using an invocation grader supported by the pinned Vally version.

1 resolved since last review

Review effort: Max (adaptive)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI adaptive review requested due to automatic review settings November 18, 2026 02:14
@github-actions github-actions Bot added awaiting-automation and removed ready-for-review Submission passed intake validation and is ready for maintainer review labels Nov 18, 2026
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@github-actions github-actions Bot added awaiting-automation ready-for-review Submission passed intake validation and is ready for maintainer review and removed ready-for-review Submission passed intake validation and is ready for maintainer review awaiting-automation labels Nov 18, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The graders reject valid indented or shell-tagged code fences and ADR title capitalization while accepting incorrectly scoped EditorConfig settings, making evaluation results unreliable.

6 open issues

Review effort: Max (adaptive)

Comment on lines +48 to +56
pattern: '(?m)^charset\s*=\s*utf-8\s*$'
- type: file-matches
config:
path: .editorconfig
pattern: '(?m)^end_of_line\s*=\s*lf\s*$'
- type: file-matches
config:
path: .editorconfig
pattern: '(?m)^insert_final_newline\s*=\s*true\s*$'
Comment thread .github/workflows/skill-eval.yml Resolved
Copilot AI adaptive review requested due to automatic review settings November 18, 2026 19:59

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The grader rejects valid list-indented shell blocks and the shell language tag, causing compliant outputs to fail at the 1.0 threshold.

5 open issues
1 resolved since last review

Review effort: Balanced (adaptive)

Adopt github/evals-action with baseline and candidate runs for affected
skills, a sticky comparison comment, a root .vally.yaml with smoke/full
suites, and completed/skill-invocation graders on every stimulus.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI adaptive review requested due to automatic review settings November 19, 2026 01:18
@github-actions github-actions Bot added awaiting-automation and removed ready-for-review Submission passed intake validation and is ready for maintainer review labels Nov 19, 2026
@github-actions github-actions Bot added ready-for-review Submission passed intake validation and is ready for maintainer review and removed awaiting-automation labels Nov 19, 2026
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@github-actions github-actions Bot added awaiting-automation and removed ready-for-review Submission passed intake validation and is ready for maintainer review labels Nov 19, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Evals: failed

Selection: Native eval paths or tags

Run: failed - 4 trials - Evidence

Baseline: failed - 4 trials - Evidence

Diagnostic: native-verdict-failed.

Eval Trials Result
convert-plaintext-to-md 1 failed
create-architectural-decision-record 1 passed
editorconfig 1 passed
update-markdown-file-index 1 passed

Quality & token summary

Eval Model Verdict Score Turns Duration Tokens
convert-plaintext-to-md claude-sonnet-5.5 failed 0.89 (threshold 1.00) 4 7149ms 81258 tokens (80536 in / 722 out)
create-architectural-decision-record claude-sonnet-5.5 passed 1.00 (threshold 1.00) 3 9896ms 55209 tokens (53980 in / 1229 out)
editorconfig claude-sonnet-5.5 passed 1.00 (threshold 1.00) 3 6743ms 54159 tokens (53311 in / 848 out)
update-markdown-file-index claude-sonnet-5.5 passed 1.00 (threshold 1.00) 5 6934ms 89238 tokens (88615 in / 623 out)

Baseline vs. Candidate comparison

Metric Baseline Candidate Delta
Outcome failed failed unchanged
Passed 3 3 0
Failed 1 1 0
Turns 14 15 +1
Duration 30200ms 30722ms +522ms (+1.7%)
Tokens 261517 tokens (258138 in / 3379 out) 279864 tokens (276442 in / 3422 out) +18347 tokens (+7.0%; out: +43 / +1.3%)
Eval Baseline Candidate Score Delta Turns Delta Duration Delta Token Delta Transition
convert-plaintext-to-md failed (0.89) failed (0.89) 0.00 0 -71ms (-1.0%) +63 tokens (+0.1%; out: +7 / +1.0%) increased-tokens
create-architectural-decision-record passed (1.00) passed (1.00) 0.00 0 +102ms (+1.0%) -41 tokens (-0.1%; out: -54 / -4.2%) reduced-tokens
editorconfig passed (1.00) passed (1.00) 0.00 0 -3ms (-0.0%) +41 tokens (+0.1%; out: +34 / +4.2%) increased-tokens
update-markdown-file-index passed (1.00) passed (1.00) 0.00 +1 +494ms (+7.7%) +18284 tokens (+25.8%; out: +56 / +9.9%) increased-tokens

Exact revision | Workflow run

@github-actions github-actions Bot added ready-for-review Submission passed intake validation and is ready for maintainer review and removed awaiting-automation labels Nov 19, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The custom concurrency group causes every PR reporting attempt to fail, while the grader rejects valid indented Markdown code blocks and falsely fails the eval.

5 open issues
1 resolved since last review

Review effort: Balanced (adaptive)

Comment thread .github/workflows/skill-eval.yml Resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The graders reject valid nested shell blocks and can award full credit when required EditorConfig settings apply only to Markdown files rather than all files.

4 open issues
1 resolved since last review

Review effort: Balanced (adaptive)

This branch was successfully deployed

1 active deployment
github-pages — 54ff8abc Deployed Nov 19, 2026 by aaronpowell via deploy #1559
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-risk:high needs-review:MEDIUM Contributor reputation check flagged MEDIUM risk new-submission PR adds at least one new contribution ready-for-review Submission passed intake validation and is ready for maintainer review workflow PR touches workflow automation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants