bai-uipath

test(uipath-agents): stage coded task fixtures instead of pasting files in prompts (#1516) Twelve uipath-agents/coded coder_eval tasks embedded full project files in initial_prompt and told the agent to recreate them verbatim, letting it pass by transcription rather than by using the skill. Move each project into _fixtures/ staged via pre_run, and rewrite the prompts to reference the on-disk project. Success criteria are unchanged, so grading is identical. Fixture staging is aligned to each checker's working directory: root-based checkers receive files at the workspace root, subdir-based checkers receive a matching named project directory. Antipattern and diagnose tasks keep their stated symptom. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

test(uipath-maestro-flow): add live Jira create/get e2e tasks (#2013) * test(uipath-maestro-flow): add live Jira create/get e2e tasks Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(uipath-maestro-flow): make Jira e2e tasks self-contained Drop the shared _jira_shared fixture; each task folder now ships its own seed/teardown/check and a small jira_is.py wrapping uip is resources run. Re-validated: both tasks 2/2, score 1.000, zero orphans. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(uipath-maestro-flow): move Jira e2e tasks into e2e/ Relocate jira_create_issue / jira_get_issue out of connector_features/ (owned solely by @baishalighosh via CODEOWNERS last-match) into e2e/, where the flow-team ownership on /tests/tasks/uipath-maestro-flow/ (incl @rockymadden) applies. Update pre/post_run hook paths to the new location. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

fix(ci): preinstall uip CLI + all tools from @dev, not the retired @alpha tag (#2055) * fix(ci): install uip CLI from @dev, not the retired @alpha tag The cli's `alpha` dist-tag was renamed to `dev` in cli PR #2650 (2026-06-26). `alpha` is now frozen at 1.197.0-alpha.20260626, so the eval image (tests/docker/Dockerfile), the catalog refresh, and the run-coder-eval dispatch were all installing a ~3-week-stale CLI. Point them at `dev` — the live prerelease train published on every push to cli/main. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * style(ci): keep cli_version dispatch description terse Revert the expanded description; workflow_dispatch renders it in a narrow column. Rationale lives in the PR body. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ci): preinstall uip CLI + all tools from @dev via npm `dev` is a publish dist-tag, not a release channel (cli PR #2650), so a `-dev` CLI's `uip tools install` resolves to `stable` and can't find a matching `-dev` tool ("No compatible version"). Install the CLI and every whitelisted tool with raw `npm install -g @uipath/<pkg>@dev` instead: npm resolves the dist-tag directly and the CLI discovers the plugins on disk by scanning $(npm root -g)/@uipath/*. Pre-install the full tool set (all 28 with a -dev build; model-hub-tool has none) and disable version-sync so no eval-time auto-install or channel resolution runs. RPA tools also come from @dev now that they ship a matching 1.199.x-dev build. The catalog refresh installs tools the same way, at the dist-tag matching the running CLI. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(ci): align catalog guard tests with npm @dev tool install install_all_tools now installs via `npm install -g <scoped>@<dist-tag>` instead of `uip tools install`, and tool_dist_tag() reads `uip --version`. Update the guard tests to assert the npm command + dist-tag, and add coverage for tool_dist_tag() derivation (dev/preview/none). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: fix maestro-case pytest duplicate-basename collision Two `test_checkers.py` files (athena_cm_event/, golden_rebuild/cm_golden_expense/) share a basename with no package markers, so `pytest tests/tasks/uipath-maestro-case/` aborts with 'import file mismatch'. Pre-existing on main; surfaced here because touching scripts/** triggers the full test-helpers.yml matrix. Add __init__.py so each gets a unique dotted module path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ci): discover eval tools dynamically at the derived dist-tag Replace the hand-copied tool whitelist in the eval Dockerfile with uip tools search discovery so it can't drift behind the CLI's TOOLS_WHITELIST, and install tools (Dockerfile + Helm/RPA host) at the dist-tag derived from CLI_VERSION rather than the literal version, so an exact CLI pin no longer fails to resolve the date-stamped rpa-tool build. Add a MIN_TOOLS floor, remove the token-bearing npmrc inside the RUN, and rewrite the stale two-feed install comment. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: bump coder_eval pin to 0.8.5 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: drop redundant maestro-case __init__.py (main fixed collision via rename in #2120) The pytest import-file-mismatch these package markers guarded against is now resolved on main by #2120, which renamed cm_golden_expense/test_checkers.py to a unique basename. Only one test_checkers.py remains repo-wide and no other task dir carries an __init__.py, so these three are dead weight. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

test(uipath-maestro-flow): restore jira_lifecycle loop+switch e2e + enable jira_search_triage (#2031) * test(uipath-maestro-flow): add live Jira loop+switch lifecycle e2e task Composite loop-and-decision Flow over the Atlassian Jira connector, graded live via `flow debug` against the Jira sandbox and a tenant re-read: the loop must create every seeded issue and the branch node route each to the correct per-priority Add-Comment. Seeded batch + markers are unique per run, so a fabricated or hardcoded output cannot pass; post_run deletes every created issue by key. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(uipath-maestro-flow): replace Jira lifecycle e2e with input-driven triage task Supersede the loop+switch lifecycle task with a harder, input-driven triage flow: a manual-trigger Flow whose trigger takes an escalation-priority input, loops a seeded High/Medium/Low batch, creates each issue, and — only for the issues whose priority equals the input — transitions to In Progress, assigns, and comments. Graded live: debug runs with the input set to "Medium" (unknown to the agent) and the tenant re-read asserts exactly the Medium issue was escalated and High/Low were untouched, so a hardcoded priority fails. The lifecycle task is a strict subset of this, so it is removed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(uipath-maestro-flow): add disabled JQL-search triage frontier task A search-then-comment flow that builds and validates (the connector exposes a Search-Issues-by-JQL activity) but cannot complete `flow debug` here: JQL search is outside the connection's curated scope (401). Shipped as *.yaml.disabled with a header documenting the gap and the one-step flip-to-pass (grant the connection search scope, then rename). Seed/check/teardown are complete so it passes as-is once scope exists. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(uipath-maestro-flow): restore jira_lifecycle, enable jira_search_triage, drop jira_triage Reshape the Jira multi-node e2e coverage on this branch: - Restore jira_lifecycle (loop a seeded batch -> per item create issue -> switch on priority -> branch-specific add-comment) as the loop+switch task, and remove the input-driven jira_triage task that had superseded it. - Enable jira_search_triage (JQL search -> loop -> add-comment): the search op is now in the connection's scope, so the task no longer 401s at debug. Strip the "disabled / expected to fail" commentary from its yaml and helper scripts now that it is discovered by the runner. Both checks run flow debug with a generous 600s cap (loop flows make several sequential live Jira calls plus Maestro debug overhead). Verified locally on Bedrock (codereval tenant): - jira_lifecycle: SUCCESS, 2/2 criteria, weighted score 1.000 (created 3 issues, each with the correct branch comment, zero orphans). - jira_search_triage: FAILURE, weighted score 0.286 (validate passes, flow debug faults). The search node itself completes and returns the matches under output.response; the flow faults at the loop because the agent binds the loop collection to output.issues (the raw Jira REST shape) instead of the connector's output.response envelope -> [400008] cannot evaluate the input collection. This is a flow-authoring pitfall the eval correctly catches, not a runtime/backend limitation, so the task stays enabled. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(uipath-maestro-flow): raise flow-debug timeouts for jira create/get e2e Give the single-node Jira e2e checks generous headroom so a slow-but-correct flow debug is not killed mid-run: run_debug 240 -> 480, criterion timeout 600 -> 900, task_timeout 1500 -> 1800. Matches the loop tasks' timeout bump. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

test(maestro-flow): cap simulated dialogs at 8 turns and stop on agent completion (#2274) * test(maestro-flow): cap simulated dialogs at 8 turns and stop on agent completion The simulated interactive flow tasks terminated on the simulator's own satisfaction, so a flow that was already built and validated kept getting pushed into re-checks and rebuilds. Three tests accounted for 15% of nightly eval cost, the top one 9%. - simulation.max_turns -> 8 (was 12-18; measured dialogs need ~4-5 rounds) - stop the dialog the first time the agent declares the work done, instead of when the simulator decides it is satisfied - the persona now volunteers the outcome, the Maestro Flow modality, and the validate-before-sign-off expectation in its opening message; every design detail stays withheld until asked Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * revert: leave solution_select simulated task untouched solution_select is a narrow single-question probe on the AskUserQuestion gate, already capped at 6 dialog rounds. It is not one of the expensive tasks this PR targets, so keep it out of the change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore(test): drop dead simulation.parallel_trials from simulated flow tasks coder_eval removed SimulationConfig.parallel_trials in v0.8.10 (nothing ever read it — trial concurrency is --max-parallel's job). SimulationConfig forbids extra keys, so these YAMLs would fail validation the moment tests/.coder-eval-version moves off 0.8.8. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(test): gate simulated-dialog stop on elicitation, not first done-claim The stop constraint fired on the agent's first "built and validated" report, which on cli_dice_roller landed at turn 1 — before the simulator revealed any withheld fact. Two gating criteria fell out of that: the "build it with the uip CLI, not by hand-editing JSON" requirement never surfaced, so the agent hand-wrote project.uiproj and left a single-nested flow that `uip maestro flow debug` refuses to run ("Flow debug requires this Flow to be inside a solution."), and the elicited project name never surfaced either. Measured against last night's canonical nightly (2026-07-24_04-33-37, same agent and model — codex/gpt-5.6-terra, framework 0.8.8): 7/8 untuned vs 4/8 with the previous wording. The stop now requires the withheld list to be discharged before it can fire, while still refusing polish, re-checks, alternative designs and rebuilds once the agent has declared done. The 8-round cap is unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(test): bundle simulated-dialog elicitation into one turn, ask for a project Two measured problems with the previous iteration, both against last night's canonical nightly (2026-07-24_04-33-37, codex/gpt-5.6-terra, 7/8, $84.54, 55 rounds): Asking one withheld fact per turn exhausted the 8-round cap on 4 of 8 tasks and truncated the agent mid-edit — expense_approval kept a correct layout and name but dropped 0.90 -> 0.40 because max_turns cut it off while it was still wiring the HITL schema. The simulator now delivers every outstanding withheld fact in a single message on the agent's first done-claim, which bounds the dialog to build -> declare -> one correction -> declare. Separately, the opening message described the deliverable as "a .flow file in the workspace", and the agent obliged literally: single-nested <Project>/<Project>.flow with no parent solution, so `uip maestro flow debug` refused ("Flow debug requires this Flow to be inside a solution.") and every runtime assertion died. The opener now asks for a Maestro Flow *project* and is explicitly barred from naming a file or path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(maestro-flow): drop the build-method demand from cli_dice_roller The persona required the project to be "built via the uip CLI ... not by hand-editing JSON", but no success criterion grades how it was built — the three checks are validate, flow debug, and the elicited project name. The demand only ever acted indirectly, and it is unsatisfiable wherever `uip maestro flow init` cannot scaffold a parent solution, which turns an environment defect into a task failure. The persona now states it has no opinion on build method, and the withheld list drops the corresponding bullet. Criteria are untouched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(test): put graded requirements in the opener, stop the dialog at first build The previous stop rule handed the agent every withheld fact in a mandated post-build correction turn. That gifted elicitation instead of testing it — zero agent-initiated questions across all 8 tasks — and paid for a second full build every time. Split each persona's fact list by what the grader actually gates on. Business intent that a gating criterion checks now goes in the opening message, phrased the way a non-technical sponsor would say it, so a build-first agent passes without a correction round. Only non-gating detail stays withheld — project name, weather data source, IxP model, SharePoint folder, approver group, and (for triage) the Sev2 policy, which no seeded case grades. The dialog now ends at the first build report, with one exception: if the agent built something that is not a Maestro Flow at all, push back, otherwise the validate criterion fails as "no .flow file found" for the wrong reason. Grading invariants are unchanged: max_turns 8, stop_on_criteria_pass false, check_criteria end_of_dialog. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(test): keep the expense-approval record inside the flow Moving "logs the outcome" into the opener dropped the cue that the final step reads what the manager entered. Asked where to log, the agent proposed UiPath Data Service and the persona accepted, so the flow had no step reading the review output and the HITL criterion failed on the missing $vars.<hitl>.output downstream read. State in the opener that the last step logs what the manager entered and keeps that record in the flow, and decline an external store if the agent offers one before building. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(test): put the ixp no-live-connection fact and SAP posting in the opener The gated criterion (weight 6.0) requires an HTTP action node for the SAP hand-off, but the fact that decides whether the agent builds one was withheld. Measured on codex/gpt-5.6-terra (sim-tuning-codex-9-postmerge): the agent spent three of four rounds hunting for live SharePoint and SAP connections that do not exist in the tenant, learned only at turn 4 that none are hooked up, and then wired core.logic.mock in place of the SAP post -- score 0.40, gating fail "no HTTP action node". Per this PR's own rule, anything a gating criterion checks has to be knowable before the first build. The opener now states that IT has not connected SharePoint or SAP in this environment, that nothing can run, and that the invoice data gets posted to a web address IT supplied. Two constraints stop the agent stalling on IT and refuse a placeholder in place of the posting step. Project name, the "New Vendor" folder and the IxP "Invoice Model" stay withheld -- the gating check greps loosely for sharepoint/uipath.ixp, so none of them gate. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(maestro-flow): raise the dice runtime-check debug budget 240s -> 600s Cloud `uip maestro flow debug` on this task regularly needs more than the 240s that check_dice_runs.py allowed: 2 of 3 codex runs died with subprocess.TimeoutExpired inside run_debug, so the weight-2.0 runtime criterion was scoring 0 on runner latency rather than on the flow the agent built. The criterion's own ceiling was 300s, leaving only 60s of headroom above the inner budget. run_debug now gets 600s and the criterion 660s. Same class of fix as #2306 (HITL completed-port smoke, turn_timeout 600 -> 900). Note the untuned arm fails this task too, for an unrelated reason -- the agent left three competing projects and the checker refused to guess. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

test(tasks): drop sandbox.driver tempdir overrides + gate against reintroduction (#2210) The Linux nightly slice runs under driver: docker (the skills-image bakes the uip CLI + every tool plugin); tasks that pinned driver: tempdir instead ran on the toolless host, so any tool-backed command (e.g. uip maestro flow validate, which needs @uipath/maestro-tool) failed with "No compatible version found" and scored zero. The Windows slice selects by the windows tag and forces --driver tempdir on the CLI, so no task needs a YAML driver override. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: gate against task YAMLs pinning sandbox.driver tempdir Deterministic full-scan check that fails the build if any tests/tasks YAML sets sandbox.driver: tempdir, so the footgun cannot return. driver: docker pins are reported as an informational note (harmless — they match the Linux default). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(tests): take the eval model from env vars, bump coder_eval to 0.9.6 (#2419) * feat(tests): take the eval model from env vars, bump coder_eval to 0.9.2 The six experiment YAMLs no longer pin `model: claude-sonnet-5`. In CI those lines were already dead — every runner passes `--model`, which outranks the experiment — and a shared file cannot name one model when the same experiment runs under codex and antigravity too. coder_eval interpolates env vars only in path fields, so the pins are removed rather than templated. The three gates that ran without `--model` now pass one, resolved from `vars.CLAUDE_CODE_MODEL` with the pinned value as the fallback, so an unset variable keeps today's behavior. run-coder-eval.yml picks up per-harness variables the same way, and antigravity's default moves to gemini-3.6-flash. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(ci): drop every hardcoded model default from the resolution path Each harness already read its own env var; the literal fallback behind it meant a missing or misspelled variable ran a whole nightly on a model nobody chose and reported it as the real one. Remove the literals so the variable is the only source and an unresolved model is a hard error. daily.sh and daily-windows.ps1 abort naming the variable they wanted; harness_smoke_model raises, reddening that harness's pre-flight line instead of smoking the wrong model; eval-runner run --model is now required and run_tests takes no default. test_daily_model keeps a denylist of the retired ids so a literal cannot creep back into any of the three mirrors. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(ci): fail the smoke gate closed when no task runs The model guard added with the env-var switch sat as the first line of the smoke step, which carries continue-on-error, and both threshold steps read zero tasks as a pass. An unresolved CLAUDE_CODE_MODEL therefore ran nothing and reported green (run 30672478895 on this branch did exactly that). Hoist the guard into its own step in both smoke workflows so an unset variable fails before any task runs, and make the Linux pass-rate gate fail closed when the smoke step failed without producing a single task.json. A legitimate empty selection still passes: that path leaves the smoke step successful. The RPA score gate already failed closed on zero tasks, which is why it went red where the Linux gate went green. Also collapses the six identical five-line comment blocks in the experiment YAMLs to one line each. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore(tests): bump coder_eval pin to 0.9.6 Picks up the Path-to-GA evalboard fixes (still-tagged tasks only, scored on runs that executed), glob patterns in criterion path fields, and the cli_called flag-parsing corrections. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

test(activation): cover the 5 uncovered skills + top up solution (#1542) * test(activation): cover the 5 uncovered skills + top up solution Add activation positives for uipath-admin (52), uipath-api-workflow (48), uipath-mcp-servers (42), uipath-automation-discovery (32), and uipath-ixp (30), each wired into activation.yaml as a stacked skill_triggered criterion. Every skill in the repo now has activation coverage (22 skill criteria + shared negatives). Top up uipath-solution positives 11 -> 24. Audit negatives: reword negative-045 (generic "UiPath Document Understanding" collided with the new uipath-ixp skill) to unambiguous classic DU / AI Center, and add adversarial negatives that sit close to the new skills' domains but must not fire them (corporate AD / Okta SSO, CNCF Serverless Workflow spec, Claude Desktop / FastMCP server, Slack summary, Tesseract OCR). The new skills have no measured recall baseline yet, so the per-skill activation-gate SKIPs them until a full activation run sets their numbers in activation_gate.py. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(activation): baseline the 6 newly-covered skills Measure recall.yes for the six skills added in this PR over their full positive sets on claude-sonnet-4-6 via Bedrock and wire the rounded baselines into activation_gate.py: automation-discovery / ixp / mcp-servers 100, solution 95, admin 90, api-workflow 85. All 22 skills now have a per-skill gate baseline; none are SKIPped. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(activation): re-baseline all 22 skills on full positive sets Measure recall.yes for every skill over its full positive set on claude-sonnet-4-6 via Bedrock (the model + full-set measurement the gate runs); all land 0.88-1.00. Set baselines to the rounded pass rate (90-100) and lower DROP_PP 15 -> 10 so each skill's gate floor sits ~10% below its measured recall. Replaces the stale 2026-05-08 baselines, several of which (review 20, maestro-case 45, agents 55) had drifted far below current performance and left the gate toothless. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(activation): clean up mislabeled and dead activation positives Drop three positives whose true intent is auth, not the labeled domain skill: agents-008 (uip login invalid_client), coded-apps-028 (base-URL 401s), coded-apps-049 (external-app redirect URI). Reword four that lacked a skill signal or carried a competing one: admin-041 -> IP allowlist / enforcement, api-workflow-016 and -021 -> explicit Workflow.json, agents-029 -> LangGraph coded agent rebuild. Defensibly ambiguous rows (external-OAuth2 -> admin, HITL-in-agent -> agents, IS-activity -> api-workflow) are kept by design. Every activation file stays >=20 rows. Removing or fixing failing rows only raises recall.yes, so the gate baselines remain valid floors. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(ci): run coder-eval through the published action, and publish an expiring evalboard link (#2942) * feat(ci): run coder-eval through the published action and publish an expiring evalboard link Three things, all in the Linux job of the dispatch workflow. **Dogfood the action.** `UiPath/coder_eval@v0` had no real consumers: it was exercised only by its own repo's self-test, so a bad release would surface there rather than in someone's workflow. This job now installs and invokes coder-eval through it. What blocked adoption before is fixed upstream: the suite lives in `tests/`, needs an agent extra, and for delegate-sdk needs a plugin inside the environment the CLI actually runs from, and none of that was expressible. The single 150-line run step becomes prep + invoke. `working-directory:` is illegal on a `uses:` step and a job-level `defaults.run` does not reach inside a composite action, so everything conditional resolves into plain inputs first. The prep body is the head of the old step moved rather than rewritten: each `export FOO=bar` became a line in an env-block accumulator, and the trailing `coder-eval run` is gone. Every guard is preserved and was exercised locally against the extracted script: the three `:?` model hard-failures, delegate's `-j` clamp with its notice, its per-key `~/.uipath/.auth` validation, and the non-empty-only re-export of the three `*_VAR` knobs. All twenty non-`_VAR` env entries are forwarded, including the four model variables the shell also reads, because coder-eval's docker driver passes `CODEX_MODEL` and `ANTIGRAVITY_MODEL` into task containers by default. The bracketed `-D sandbox.docker.env_passthrough_extra=[...]` override now goes through the action's `args` input, which appends one argument per line verbatim. That is the same protection the old array-and-quote gave it, for the same reason: to bash the list is a character class. **An evalboard link per run.** The run uploads to a dedicated `runs-gha` container, which expires it after 14 days, and the link lands in the job summary. Entirely best-effort: every step warns and returns 0, because a missing dashboard link must never be mistaken for a failed eval. Inert until `AZURE_EVAL_UPLOAD_CLIENT_ID` exists as a repo variable, so this behaves exactly as before until the Azure identity is provisioned. Credentials are OIDC, not a storage key: `workflow_dispatch` runs the workflow definition from whatever branch the dispatcher picks, so a key in a repo secret would let any branch delete months of nightly history on that account. **Two fixes to things already broken.** The workflow had no redaction step, unlike smoke-skills.yml, so its artifact has been shipping unredacted transcripts; it is ported into both jobs and covers every secret this job forwards, not just the three smoke-skills strips. And the Linux artifact globs were off by one directory level, uploading no experiment- or variant-level report at all, because `--run-dir /tmp/runs` makes that directory itself the run dir. `run-name` is one line and independently the highest value per character available here: all ~2,500 historical runs are titled "Run Coder Eval". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor(ci): authenticate the evalboard upload with the existing storage credential The evalboard upload minted a GitHub OIDC token and exchanged it for an Azure credential, which needed a new Entra app registration, a federated credential scoped to a GitHub environment, and a container role assignment before it could work at all. The ADO pipelines already authenticate to this same storage account with `AZURE_STORAGE_KEY` / `AZURE_STORAGE_ACCOUNT`, and eval-runner passes that value straight to `BlobServiceClient`, so reusing it removes every one of those provisioning steps: two repo settings and nothing else. The Azure SDK sniffs the credential string, so the secret holds either the account key or a container-scoped SAS token with no change here. The SAS is preferable, since an account key is authority over the whole account including the nightly-history container, and this workflow runs its definition from whatever branch the dispatcher picks. Drops `id-token: write`, the `eval-upload` environment and the token-minting step. The gate moves to a job-env boolean because the `secrets` context is unavailable in any `if:`; deriving a boolean rather than exporting the secret keeps the credential in the upload step alone, matching how ADO scopes it. The container name is now hardcoded, since the unlisted evalboard source and the 14-day expiry rule both depend on ad-hoc runs never landing in `runs`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor(ci): pass the whole coder-eval command line through the action's args Follows the action's input simplification: it no longer promotes any of `coder-eval run`'s flags to named inputs, so the task globs and `--model` join the `-e`, `--type`, `-j`, `-v` and `-D` entries in the single `args` block the prep step already composed. The `model` step output goes away with them. Task globs are word-split when appended (TASK_GLOBS is a space-separated list from the partition job) but never pathname-expanded, which is an improvement on what the action used to do with them: the CLI expands them itself, so `**` works without `globstar` and a glob matching nothing exits 1 rather than reaching the CLI as a literal path. `step-summary: "false"` and `junit-path` are gone. The action writes no job summary at all now and reports `run-md-path` instead, so the append can happen after redaction, which is the order this workflow needs and previously had to work around. The append reads that output with the literal path as a fallback, because a composite action's outputs may not propagate from a step that exited non-zero and this step runs on failure. The JUnit report moves to /tmp/runs/junit.xml, derived from run-dir; nothing consumed it at the old path. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(ci): temporarily pin the action to the PR-147 head for an e2e dry run The `v0` tag still serves the ten-input action, so this workflow's `with:` block would have `args`, `extras`, `extra-packages` and `working-directory` silently dropped — GitHub ignores an input the referenced tag does not declare. Pinning the PR head validates the whole path before anything merges and before Release moves `v0`. REVERT TO @v0 BEFORE MERGING. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(ci): trim the comment bulk around the action split Comments only. Fifteen blocks cut to their contract, 118 lines out for 58 in, so the file sits at its existing 33% comment density instead of above it. Also drops a stale claim: the two-step-split banner said TASK_GLOBS "is handed straight to the action's `tasks` input, which word-splits it". There is no `tasks` input, and `args` entries are passed verbatim. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(ci): put the evalboard link at the top of the job summary The link was appended after run.md, which is roughly 40 lines of tables, so it landed below the fold and was easy to miss. Prepend rather than reorder the steps: the evalboard step moves /tmp/runs, which both the artifact upload and the verdict step read, so it has to stay last. $GITHUB_STEP_SUMMARY is an ordinary file, so the block is written to a temp file with the existing summary appended, then truncate-and-rewritten in place to keep the inode the runner tracks. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(ci): publish evalboard before the report so the link renders first The previous attempt prepended to $GITHUB_STEP_SUMMARY inside the evalboard step. That cannot work: the variable points at a SEPARATE file per step, and GitHub builds the job summary by concatenating them in step order, so the step read its own empty file and its block still rendered last. Verified on run 33564007553, where the link stayed at the bottom. Move the publish above the step that appends run.md, which is the only way to control position, and copy the run directory instead of moving it so the artifact upload and the verdict still find /tmp/runs. Also emit a ::notice::, which renders above the job list on the run page. Keep "Run Coder Eval" as the run-name prefix so the run list stays scannable by workflow as well as by glob, agent and actor. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(ci): grade the run directory, not its newest variant `--run-dir /tmp/runs` makes /tmp/runs the run dir itself, so its subdirectories are VARIANTS. The verdict step selected one with `ls -td /tmp/runs/*/ | head -n 1`, which is correct only while nightly.yaml declares a single variant: a second one would silently grade a single arm and could report a red run green. This is the same off-by-one the artifact globs in this branch already fix, one step lower. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * ci(coder-eval): consume the published action at @v0 The temporary SHA pin pointed at the PR branch of coder_eval#147, which is now merged and released as 0.11.6 with v0 promoted to it, so the eight-input surface this workflow passes is what v0 serves. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

ci: pin coder_eval to a published version (wheel + GHCR image) (#1467) * ci: pin coder_eval to exact version via skills manifest Add coderEvalVersion (v0.5.0) to version-manifest.json as the single source of truth for which coder_eval release the suite runs against. Resolve it in run-coder-eval.yml (once in the partition job, passed to both run jobs; the coder_eval_ref input still overrides) and in the smoke-skills.yml e2e runner. Replaces tracking coder_eval's default branch. The advisory validate-task-schema job stays on main by design (forward-looking schema canary). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ci: move coder_eval pin to tests/.coder-eval-version Keep the engine pin out of product version metadata — it's test infra, so it lives next to the tasks it's bound to. run-coder-eval.yml and smoke-skills.yml read the tag from tests/.coder-eval-version; the coder_eval_ref input still overrides. Shorten the input description for the small workflow_dispatch box. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ci: rename coder_eval_ref input to coder_eval_version It's a pinned version now, not a generic ref. Renames the workflow_dispatch input, the partition job output, and the resolve step output to match. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ci: keep coder_eval_ref naming (revert the version rename) The override accepts any git ref — branch, tag, or SHA — not only a version. The pinned default lives in tests/.coder-eval-version, so the knob itself is a ref. Reverts the coder_eval_version rename. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ci: bump coder_eval pin to v0.6.1 v0.6.1 pins the Claude Code CLI binary in the agent image (UiPath/coder_eval#425), making the agent version deterministic. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: pin remaining coder_eval checkouts to the skills version manifest RPA smoke (smoke-rpa-skills), the activation gate (activation-gate), and the advisory task-schema validate (smoke-skills) still checked out coder_eval@main, so they drifted from the engine the skills smoke, ad-hoc, and nightly paths run. Pin all three to tests/.coder-eval-version so every coder_eval consumer references the same version. RPA smoke reads the pin under shell: bash since windows-latest defaults to PowerShell (no `tr`). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: own the coder_eval version pin in CODEOWNERS tests/.coder-eval-version is the single source of truth for the engine version that smoke, ad-hoc, and nightly all run against, but it only matched the default owners — so a bump could be approved without the eval-CI team. Route it to the same owners as the /.github/ workflows that consume it, so a version change is reviewed by the people who own the eval engine. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: drop verbose comment from coder_eval pin CODEOWNERS entry Keep the pin owned by the eval-CI team, just place the rule directly under the /.github/ entry without the explanatory block. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: consume coder_eval as pinned wheel + GHCR image, not source Pin coder_eval to 0.6.2 and have every consumer (PR smoke, RPA smoke, the advisory schema gate, the ad-hoc dispatch, and the activation gate) install the host CLI from the published wheel `coder-eval==<pin>` (Azure Artifacts coder_eval feed + ml-packages for deps) instead of checking out and installing coder_eval source. The Docker-driver jobs additionally pull `ghcr.io/uipath/coder-eval-agent:<pin>` from GHCR rather than building the agent image with `make docker-image`. The pin (tests/.coder-eval-version) now holds the bare version, consumed as both the wheel version and the image tag. The dispatch input `coder_eval_ref` becomes `coder_eval_version` (a published release to dry-run before bumping the pin). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: trim coder_eval_version workflow input description Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>