You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
My contribution adds a new instruction, prompt, agent, skill, workflow, or canvas extension file in the correct directory.
The file follows the required naming convention.
The content is clearly structured and follows the example format.
I have tested my instructions, prompt, agent, skill, workflow, or canvas extension with GitHub Copilot.
I have run npm start and verified that README.md is up to date.
I am targeting the main branch for this pull request.
Description
Pilot behavioral evaluation for convert-plaintext-to-md so skill changes can be assessed for behavior as well as structure. The read-only Vally workflow lints affected eval specifications alongside skills, including on fork PRs. A separate advisory workflow runs the behavioral eval on trusted changes and publishes a job summary and report artifact; the existing Submission Gate comment will report its status once the gate configuration is on main. The eval pipeline preserves Vally's exit status while logging its output.
Type of Contribution
New instruction file.
New prompt file.
New agent file.
New plugin.
New skill file.
New agentic workflow.
New canvas extension.
Update to existing instruction, prompt, agent, plugin, skill, workflow, or canvas extension.
Other (please specify): GitHub Actions workflow and skill eval specification.
Additional Notes
node --test .github/scripts/review/submission-gate.test.mjs: passed (50 tests), including a regression test for successful and failing Vally commands.
go run github.com/rhysd/actionlint/cmd/actionlint@latest .github/workflows/skill-check.yml .github/workflows/skill-check-comment.yml .github/workflows/skill-eval.yml: failed because actionlint does not recognize the copilot-requests permission.
go run github.com/rhysd/actionlint/cmd/actionlint@latest -ignore 'unknown permission scope "copilot-requests"' .github/workflows/skill-eval.yml: passed.
The latest live eval run correctly reports failure: six of seven graders passed, but the shell-code-block grader did not. The summary and full report artifact were still published. An earlier run falsely reported success because tee masked Vally's exit code; the workflow now uses pipefail to prevent this. Behavioral evals require Copilot CLI billing and do not run on fork PRs. The grader regexes compile with Vally 0.21.0, which supports the inline flags used by the pilot.
By submitting this pull request, I confirm that my contribution abides by the Code of Conduct and will be licensed under the MIT License.
Risk tier:merge-risk:high — Privileged execution, automation, or review-policy change Required to merge: passing submission-gate checks plus 1 approval from reviewers with write access, including a maintainer with admin or maintain permission.
Why this tier
.github/scripts/review/submission-gate.test.mjs is a high-risk path (automation, scripts, MCP config, hooks, or review policy)
.github/submission-gate.yml is a high-risk path (automation, scripts, MCP config, hooks, or review policy)
.github/workflows/skill-check-comment.yml is a high-risk path (automation, scripts, MCP config, hooks, or review policy)
.github/workflows/skill-check.yml is a high-risk path (automation, scripts, MCP config, hooks, or review policy)
.github/workflows/skill-eval.yml is a high-risk path (automation, scripts, MCP config, hooks, or review policy)
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
The workflow can mask evaluation failures, while the graders can reject valid formatting, miss an omitted approval prerequisite, and conceal regressions in skill-defined output rules.
Adds a behavioral evaluation pilot for convert-plaintext-to-md, complementing structural skill validation with advisory results in the Submission Gate.
Changes:
Adds a deployment-notes fixture and graders for Markdown formatting, content preservation, and source-file preservation.
Extends read-only Vally linting and result counts to eval specifications, including fork PRs.
Adds a trusted-change evaluation workflow with job summaries, report artifacts, and advisory gate integration.
Eval-spec lint resolves the latest Vally release, while .github/workflows/skill-eval.yml:60 pins execution to 0.21.0. As new releases ship, the two checks can disagree about supported schema features or regex syntax for the same spec. Pin this lint command to the runner's version so validation stays consistent.
All graders inspect files or the source diff, so a conversion produced without activating convert-plaintext-to-md can still receive full credit. Since the prompt already supplies the graded formatting rules, the pilot can miss regressions in skill activation. Require verified activation of this skill as a scored condition, using an invocation grader supported by the pinned Vally version.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
The graders reject valid indented or shell-tagged code fences and ADR title capitalization while accepting incorrectly scoped EditorConfig settings, making evaluation results unreliable.
Adopt github/evals-action with baseline and candidate runs for affected
skills, a sticky comparison comment, a root .vally.yaml with smoke/full
suites, and completed/skill-invocation graders on every stimulus.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
The custom concurrency group causes every PR reporting attempt to fail, while the grader rejects valid indented Markdown code blocks and falsely fails the eval.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
The graders reject valid nested shell blocks and can award full credit when required EditorConfig settings apply only to Markdown files rather than all files.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request Checklist
npm startand verified thatREADME.mdis up to date.mainbranch for this pull request.Description
Pilot behavioral evaluation for
convert-plaintext-to-mdso skill changes can be assessed for behavior as well as structure. The read-only Vally workflow lints affected eval specifications alongside skills, including on fork PRs. A separate advisory workflow runs the behavioral eval on trusted changes and publishes a job summary and report artifact; the existing Submission Gate comment will report its status once the gate configuration is onmain. The eval pipeline preserves Vally's exit status while logging its output.Type of Contribution
Additional Notes
node --test .github/scripts/review/submission-gate.test.mjs: passed (50 tests), including a regression test for successful and failing Vally commands.npx --yes @microsoft/vally-cli@0.21.0 lint --eval-spec evals/skills/convert-plaintext-to-md/eval.yaml --strict: passed.npm start: passed; generated READMEs were up to date.bash .github/scripts/maintenance/fix-line-endings.sh: passed.go run github.com/rhysd/actionlint/cmd/actionlint@latest .github/workflows/skill-check.yml .github/workflows/skill-check-comment.yml .github/workflows/skill-eval.yml: failed because actionlint does not recognize thecopilot-requestspermission.go run github.com/rhysd/actionlint/cmd/actionlint@latest -ignore 'unknown permission scope "copilot-requests"' .github/workflows/skill-eval.yml: passed.The latest live eval run correctly reports failure: six of seven graders passed, but the shell-code-block grader did not. The summary and full report artifact were still published. An earlier run falsely reported success because
teemasked Vally's exit code; the workflow now usespipefailto prevent this. Behavioral evals require Copilot CLI billing and do not run on fork PRs. The grader regexes compile with Vally 0.21.0, which supports the inline flags used by the pilot.By submitting this pull request, I confirm that my contribution abides by the Code of Conduct and will be licensed under the MIT License.