bai-uipath

test(uipath-maestro-flow): brownfield edit evals on the DevCon billing flow (#3978) * test(uipath-maestro-flow): brownfield edit evals on the DevCon billing flow Three tasks edit a Flow the product wrote rather than the SDK: the BillingDisputeResolution solution exactly as the classic Flow skill built it on the 2026-09-29 nightly (inline agents with sidecars, a context index, Data Service connector nodes, a switch with per-case conditions). A shared grader compares the result with the committed fixture: every node, edge, binding, variable, definition and agent sidecar the task did not name must be unchanged, and original nodes must keep their canvas positions. The existing edit tasks grade validate and debug output only, so a round trip that drops a context index id or re-lays out the canvas passes them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(uipath-maestro-flow): keep path-to-ga on the switch-route brownfield task only The three tasks share one fixture and each preservation check covers the whole file, so one never-skip task carries the round-trip signal. The switch task also covers the open per-case switch loss (builder-sdk #1131); the other two follow maturity. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(uipath-maestro-flow): point the preservation checker usage at REFERENCE_DIR The usage line named the retired $TASK_DIR, which the task host-path gate rejects. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(uipath-maestro-flow): grade brownfield edits by running the agent's code - credit-cap: run the new Script with node at 120/500/750/10000 and require min(creditAmount, 500), instead of looking for the text "500". - switch-route: run the new case's expression for each recommended action and require it to match only request_more_info; the new End's pass-through outputs must evaluate like the other End nodes'. - writer-prompt: also require "next statement", every original prompt line, contentTokens rawString matching content, and no other agent.json change; the writer's .flow node is no longer exempt. - Preservation: compare definitions as full JSON, not just (nodeType, version). - Drop post_run: the tasks never upload a solution, and cleanup_solutions.py would delete the fixture's SolutionId on any tenant that has it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(uipath-maestro-flow): drop path-to-ga from the switch-route brownfield task The path-to-ga gate needs an allowlisted approval on every new head commit, so each push to this PR blocks on a re-review. All three brownfield tasks now follow normal maturity skipping; the tag can be added on its own later. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

test(uipath-maestro-flow): require the Flow edit tasks to go through flow merge and keep the rest of the flow (#4031) * test(uipath-maestro-flow): require brownfield .flow edits to come from flow merge The switch-route and credit-cap brownfield tasks graded only the result, so an agent that skipped the skill's decompile/compile/merge loop and hand-edited the .flow JSON scored the same as one that used it. Both now fail a hand edit: merge renumbers every edge from the compiled source, so an original edge id that survives means the file was not written by merge. switch-route also compares the original cases by id, label and behavior instead of spelling (merge requotes 'x' as "x"), and requires the switch to keep its label, which a round trip through the SDK can lose. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(uipath-maestro-flow): name the merge check's edge-renumbering dependency and point its failure at brownfield.md Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(uipath-maestro-flow): require the Flow edit tasks' .flow to come from flow merge and keep the rest of the flow The six edit/ tasks graded only validate and the debug output, so a hand-edited .flow and a merge that rewrote untouched parts of the flow both passed. - move the merged check into _shared/check_brownfield_preserved.py (--aspect merged, now also reading subflow edges) so both task families share it - preserved checker: --removable-node for nodes an edit may delete or move into a subflow; a definition no node uses any more may go - add the merged and preserved criteria (weight 3, gating) to all six edit tasks, each with the nodes and edges its edit is allowed to touch Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(tests): quote the edge specs in the add_node and move_node preserved checks run_command runs the criterion through a shell, so the unquoted '>' redirected the checker's stdout to a file and cut the edge spec in half. The test now rejects an unquoted shell operator in these commands. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

ci(tests): gate Windows-reachable shell commands for cmd.exe safety (#3145) * ci(tests): gate Windows-reachable shell commands for cmd.exe safety On Windows, coder_eval dispatches pre_run/post_run through asyncio.create_subprocess_shell and run_command criteria through subprocess.run(shell=True). Both resolve to %COMSPEC% /c, which is cmd.exe, not sh. Nothing connected "this experiment or task runs on Windows" to "therefore its commands must be cmd-safe", which is how #2756 put a multi-line POSIX sh pre_run hook into nightly.yaml and took the Windows nightly split to 7/7 status=ERROR at setup (fixed by #3116). The three surfaces fail with very different volume. A bad pre_run aborts the task because fail_on_error defaults true, which is how this one got noticed. A bad post_run only warn-logs. A bad run_command criterion silently scores 0, which in the report is indistinguishable from the agent having written bad code, so it would read as a model regression. Adds that missing link as a second gate inside the existing scripts/check-task-driver.py, which already parses the corpus and already encodes the windows-tag-to-platform relationship in its docstring. Scope is run_command criteria and hooks on windows-tagged tasks, plus hooks on any experiment classified windows-reachable. The classification table is fail-closed: an unlisted experiment fails the gate until someone decides whether it can reach a Windows runner. The scan is quote-aware. cmd treats a double-quoted run as literal text, so `python -c "import sys; sys.exit(0 if ... else 1)"` is genuinely safe despite being full of sh metacharacters. All four Windows-tagged run_command criteria in the corpus are that shape and a naive scan flags every one. `&&` and `||` are not flagged either, since cmd supports both. nightly.yaml's POSIX post_run cleanup is exempt because coder_eval_uipath's prepare_windows_experiment.py rewrites it to PowerShell before the Windows slice sees it. The exemption is exact-match, so drifting that command fails this gate and the overlay's own len(matches) != 1 assertion together rather than letting one of them age out quietly. Fixes one live instance the gate found: smoke-windows.yaml's post_run was `rm -rf node_modules .npm-prefix`, running on uipath-windows-latest and not covered by the overlay, so it warn-logged and leaked node_modules on every task. Now a pwsh one-liner. Also covers gate 1, the driver pin, which had no tests. The job name in task-driver-gate.yml is a required status check in the main ruleset, so both gates run in one step under the existing name. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(tests): cut the Windows hook gate to the single-line rule Replaces the construct-based cmd.exe-safety gate from the previous commit with just the newline rule on experiment pre_run/post_run commands. The bug this gate exists for was a multi-line `|-` block scalar, and cmd.exe parses only the first line of what it is handed, so "hooks must be a single line" catches it. Every hook command in the corpus already satisfies that, and a newline is a syntactic fact rather than a heuristic, so the rule needs no per-experiment configuration and has no false-positive surface. The removed machinery was all downstream of one choice: also catching POSIX sh that fits on one line. That needed a construct table; the construct table flagged all four Windows-tagged run_command criteria, which are `python -c "..."` payloads that cmd handles fine, so it needed quote-awareness; gating constructs then failed seven legitimately Linux-only experiments, so it needed a Windows-reachability table; and nightly's post_run cleanup is legitimately POSIX because the overlay rewrites it, so it needed an exemption table. Roughly 460 lines to pre-empt a failure mode that has not happened. What this gives up is recorded as a test rather than left implicit: test_known_gap_single_line_posix_passes asserts a one-line POSIX hook still gets through, with a comment saying what it would cost to catch. If that ever breaks a Windows run, that is the moment to add the table back. Also drops the task-level scope. Windows-tagged tasks' run_command criteria are the latent silent-zero class, not the outage, and there are currently zero offenders among them. Kept from the previous commit: the smoke-windows.yaml PowerShell fix. That is a real live bug found along the way (POSIX `rm -rf` dispatched to cmd.exe on uipath-windows-latest, warn-logging and leaking node_modules every task), and it stands independently of how much checking we keep. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(tests): trim the hook gate comments to the essentials Prose only. No behaviour change: gate output, exit codes and all 8 tests are unchanged, and the end-to-end replay of 80c757e03 still exits 1 on nightly.yaml:65. Cuts the gate-2 docstring from 35 lines to 8, the workflow comments from 16 to 4, the test module docstring from 19 to 6, and the smoke-windows.yaml note from 6 lines to 3. The known-gap test keeps its explanation, since what the gate deliberately does not catch is the one thing a reader cannot infer from the code. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(tests): make the windows smoke cleanup exit 0 when there is nothing to clean The pwsh `Remove-Item -LiteralPath a,b -ErrorAction SilentlyContinue` form exited 1 whenever either path was already absent, which is the common case, so post_run warn-logged on nearly every task. Guard each path with Test-Path and use -ErrorAction Stop so the exit code is 0 with nothing to clean and non-zero only on a real delete failure. Spelled out per path so the command contains no `$`, which POSIX sh would expand away. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(ci): extend the hook gate to windows-tagged tasks and variant hooks The newline rule only reached experiment-scope hooks, so two positions cmd.exe can also reach were unguarded: the hooks of `windows`-tagged tasks, and hooks under `variants[]` (the scope walk descended into dict values only, and `variants` is a list). Neither position holds a hook today, so the corpus count is unchanged at 11 and no existing file starts failing. Task hooks are gated on the `windows` tag rather than scanned wholesale: 24 multi-line task hooks exist, none of them windows-tagged, and one-lining a 35-line heredoc that never meets cmd.exe would be churn for no safety. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(ci): register the task gate as a required check Main's new contract guard asserts that every job in test-helpers.yml appears in docs/REQUIRED-CHECKS.md § Current target set: the workflow exists to hold cheap always-run guards, so a job there that is not required has no reason to be there. This branch's `task-gate-tests` job was the one exception after the merge, and the guard failed on it. Adds the row. The live ruleset is untouched — apply-required-checks.sh reads this table, and per § Applying a change the context should only be added after the job has reported once on a PR opened post-merge. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

test(uipath-maestro-flow): require brownfield .flow edits to come from flow merge (#4029) * test(uipath-maestro-flow): require brownfield .flow edits to come from flow merge The switch-route and credit-cap brownfield tasks graded only the result, so an agent that skipped the skill's decompile/compile/merge loop and hand-edited the .flow JSON scored the same as one that used it. Both now fail a hand edit: merge renumbers every edge from the compiled source, so an original edge id that survives means the file was not written by merge. switch-route also compares the original cases by id, label and behavior instead of spelling (merge requotes 'x' as "x"), and requires the switch to keep its label, which a round trip through the SDK can lose. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(uipath-maestro-flow): name the merge check's edge-renumbering dependency and point its failure at brownfield.md Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

test(maestro-flow): cap simulated dialogs at 8 turns and stop on agent completion (#2274) * test(maestro-flow): cap simulated dialogs at 8 turns and stop on agent completion The simulated interactive flow tasks terminated on the simulator's own satisfaction, so a flow that was already built and validated kept getting pushed into re-checks and rebuilds. Three tests accounted for 15% of nightly eval cost, the top one 9%. - simulation.max_turns -> 8 (was 12-18; measured dialogs need ~4-5 rounds) - stop the dialog the first time the agent declares the work done, instead of when the simulator decides it is satisfied - the persona now volunteers the outcome, the Maestro Flow modality, and the validate-before-sign-off expectation in its opening message; every design detail stays withheld until asked Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * revert: leave solution_select simulated task untouched solution_select is a narrow single-question probe on the AskUserQuestion gate, already capped at 6 dialog rounds. It is not one of the expensive tasks this PR targets, so keep it out of the change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore(test): drop dead simulation.parallel_trials from simulated flow tasks coder_eval removed SimulationConfig.parallel_trials in v0.8.10 (nothing ever read it — trial concurrency is --max-parallel's job). SimulationConfig forbids extra keys, so these YAMLs would fail validation the moment tests/.coder-eval-version moves off 0.8.8. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(test): gate simulated-dialog stop on elicitation, not first done-claim The stop constraint fired on the agent's first "built and validated" report, which on cli_dice_roller landed at turn 1 — before the simulator revealed any withheld fact. Two gating criteria fell out of that: the "build it with the uip CLI, not by hand-editing JSON" requirement never surfaced, so the agent hand-wrote project.uiproj and left a single-nested flow that `uip maestro flow debug` refuses to run ("Flow debug requires this Flow to be inside a solution."), and the elicited project name never surfaced either. Measured against last night's canonical nightly (2026-07-24_04-33-37, same agent and model — codex/gpt-5.6-terra, framework 0.8.8): 7/8 untuned vs 4/8 with the previous wording. The stop now requires the withheld list to be discharged before it can fire, while still refusing polish, re-checks, alternative designs and rebuilds once the agent has declared done. The 8-round cap is unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(test): bundle simulated-dialog elicitation into one turn, ask for a project Two measured problems with the previous iteration, both against last night's canonical nightly (2026-07-24_04-33-37, codex/gpt-5.6-terra, 7/8, $84.54, 55 rounds): Asking one withheld fact per turn exhausted the 8-round cap on 4 of 8 tasks and truncated the agent mid-edit — expense_approval kept a correct layout and name but dropped 0.90 -> 0.40 because max_turns cut it off while it was still wiring the HITL schema. The simulator now delivers every outstanding withheld fact in a single message on the agent's first done-claim, which bounds the dialog to build -> declare -> one correction -> declare. Separately, the opening message described the deliverable as "a .flow file in the workspace", and the agent obliged literally: single-nested <Project>/<Project>.flow with no parent solution, so `uip maestro flow debug` refused ("Flow debug requires this Flow to be inside a solution.") and every runtime assertion died. The opener now asks for a Maestro Flow *project* and is explicitly barred from naming a file or path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(maestro-flow): drop the build-method demand from cli_dice_roller The persona required the project to be "built via the uip CLI ... not by hand-editing JSON", but no success criterion grades how it was built — the three checks are validate, flow debug, and the elicited project name. The demand only ever acted indirectly, and it is unsatisfiable wherever `uip maestro flow init` cannot scaffold a parent solution, which turns an environment defect into a task failure. The persona now states it has no opinion on build method, and the withheld list drops the corresponding bullet. Criteria are untouched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(test): put graded requirements in the opener, stop the dialog at first build The previous stop rule handed the agent every withheld fact in a mandated post-build correction turn. That gifted elicitation instead of testing it — zero agent-initiated questions across all 8 tasks — and paid for a second full build every time. Split each persona's fact list by what the grader actually gates on. Business intent that a gating criterion checks now goes in the opening message, phrased the way a non-technical sponsor would say it, so a build-first agent passes without a correction round. Only non-gating detail stays withheld — project name, weather data source, IxP model, SharePoint folder, approver group, and (for triage) the Sev2 policy, which no seeded case grades. The dialog now ends at the first build report, with one exception: if the agent built something that is not a Maestro Flow at all, push back, otherwise the validate criterion fails as "no .flow file found" for the wrong reason. Grading invariants are unchanged: max_turns 8, stop_on_criteria_pass false, check_criteria end_of_dialog. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(test): keep the expense-approval record inside the flow Moving "logs the outcome" into the opener dropped the cue that the final step reads what the manager entered. Asked where to log, the agent proposed UiPath Data Service and the persona accepted, so the flow had no step reading the review output and the HITL criterion failed on the missing $vars.<hitl>.output downstream read. State in the opener that the last step logs what the manager entered and keeps that record in the flow, and decline an external store if the agent offers one before building. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(test): put the ixp no-live-connection fact and SAP posting in the opener The gated criterion (weight 6.0) requires an HTTP action node for the SAP hand-off, but the fact that decides whether the agent builds one was withheld. Measured on codex/gpt-5.6-terra (sim-tuning-codex-9-postmerge): the agent spent three of four rounds hunting for live SharePoint and SAP connections that do not exist in the tenant, learned only at turn 4 that none are hooked up, and then wired core.logic.mock in place of the SAP post -- score 0.40, gating fail "no HTTP action node". Per this PR's own rule, anything a gating criterion checks has to be knowable before the first build. The opener now states that IT has not connected SharePoint or SAP in this environment, that nothing can run, and that the invoice data gets posted to a web address IT supplied. Two constraints stop the agent stalling on IT and refuse a placeholder in place of the posting step. Project name, the "New Vendor" folder and the IxP "Invoice Model" stay withheld -- the gating check greps loosely for sharepoint/uipath.ixp, so none of them gate. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(maestro-flow): raise the dice runtime-check debug budget 240s -> 600s Cloud `uip maestro flow debug` on this task regularly needs more than the 240s that check_dice_runs.py allowed: 2 of 3 codex runs died with subprocess.TimeoutExpired inside run_debug, so the weight-2.0 runtime criterion was scoring 0 on runner latency rather than on the flow the agent built. The criterion's own ceiling was 300s, leaving only 60s of headroom above the inner budget. run_debug now gets 600s and the criterion 660s. Same class of fix as #2306 (HITL completed-port smoke, turn_timeout 600 -> 900). Note the untuned arm fails this task too, for an unrelated reason -- the agent left three competing projects and the checker refused to guess. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

feat(tests): take the eval model from env vars, bump coder_eval to 0.9.6 (#2419) * feat(tests): take the eval model from env vars, bump coder_eval to 0.9.2 The six experiment YAMLs no longer pin `model: claude-sonnet-5`. In CI those lines were already dead — every runner passes `--model`, which outranks the experiment — and a shared file cannot name one model when the same experiment runs under codex and antigravity too. coder_eval interpolates env vars only in path fields, so the pins are removed rather than templated. The three gates that ran without `--model` now pass one, resolved from `vars.CLAUDE_CODE_MODEL` with the pinned value as the fallback, so an unset variable keeps today's behavior. run-coder-eval.yml picks up per-harness variables the same way, and antigravity's default moves to gemini-3.6-flash. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(ci): drop every hardcoded model default from the resolution path Each harness already read its own env var; the literal fallback behind it meant a missing or misspelled variable ran a whole nightly on a model nobody chose and reported it as the real one. Remove the literals so the variable is the only source and an unresolved model is a hard error. daily.sh and daily-windows.ps1 abort naming the variable they wanted; harness_smoke_model raises, reddening that harness's pre-flight line instead of smoking the wrong model; eval-runner run --model is now required and run_tests takes no default. test_daily_model keeps a denylist of the retired ids so a literal cannot creep back into any of the three mirrors. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(ci): fail the smoke gate closed when no task runs The model guard added with the env-var switch sat as the first line of the smoke step, which carries continue-on-error, and both threshold steps read zero tasks as a pass. An unresolved CLAUDE_CODE_MODEL therefore ran nothing and reported green (run 30672478895 on this branch did exactly that). Hoist the guard into its own step in both smoke workflows so an unset variable fails before any task runs, and make the Linux pass-rate gate fail closed when the smoke step failed without producing a single task.json. A legitimate empty selection still passes: that path leaves the smoke step successful. The RPA score gate already failed closed on zero tasks, which is why it went red where the Linux gate went green. Also collapses the six identical five-line comment blocks in the experiment YAMLs to one line each. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore(tests): bump coder_eval pin to 0.9.6 Picks up the Path-to-GA evalboard fixes (still-tagged tasks only, scored on runs that executed), glob patterns in criterion path fields, and the cli_called flag-parsing corrections. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

feat(ci): run coder-eval through the published action, and publish an expiring evalboard link (#2942) * feat(ci): run coder-eval through the published action and publish an expiring evalboard link Three things, all in the Linux job of the dispatch workflow. **Dogfood the action.** `UiPath/coder_eval@v0` had no real consumers: it was exercised only by its own repo's self-test, so a bad release would surface there rather than in someone's workflow. This job now installs and invokes coder-eval through it. What blocked adoption before is fixed upstream: the suite lives in `tests/`, needs an agent extra, and for delegate-sdk needs a plugin inside the environment the CLI actually runs from, and none of that was expressible. The single 150-line run step becomes prep + invoke. `working-directory:` is illegal on a `uses:` step and a job-level `defaults.run` does not reach inside a composite action, so everything conditional resolves into plain inputs first. The prep body is the head of the old step moved rather than rewritten: each `export FOO=bar` became a line in an env-block accumulator, and the trailing `coder-eval run` is gone. Every guard is preserved and was exercised locally against the extracted script: the three `:?` model hard-failures, delegate's `-j` clamp with its notice, its per-key `~/.uipath/.auth` validation, and the non-empty-only re-export of the three `*_VAR` knobs. All twenty non-`_VAR` env entries are forwarded, including the four model variables the shell also reads, because coder-eval's docker driver passes `CODEX_MODEL` and `ANTIGRAVITY_MODEL` into task containers by default. The bracketed `-D sandbox.docker.env_passthrough_extra=[...]` override now goes through the action's `args` input, which appends one argument per line verbatim. That is the same protection the old array-and-quote gave it, for the same reason: to bash the list is a character class. **An evalboard link per run.** The run uploads to a dedicated `runs-gha` container, which expires it after 14 days, and the link lands in the job summary. Entirely best-effort: every step warns and returns 0, because a missing dashboard link must never be mistaken for a failed eval. Inert until `AZURE_EVAL_UPLOAD_CLIENT_ID` exists as a repo variable, so this behaves exactly as before until the Azure identity is provisioned. Credentials are OIDC, not a storage key: `workflow_dispatch` runs the workflow definition from whatever branch the dispatcher picks, so a key in a repo secret would let any branch delete months of nightly history on that account. **Two fixes to things already broken.** The workflow had no redaction step, unlike smoke-skills.yml, so its artifact has been shipping unredacted transcripts; it is ported into both jobs and covers every secret this job forwards, not just the three smoke-skills strips. And the Linux artifact globs were off by one directory level, uploading no experiment- or variant-level report at all, because `--run-dir /tmp/runs` makes that directory itself the run dir. `run-name` is one line and independently the highest value per character available here: all ~2,500 historical runs are titled "Run Coder Eval". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor(ci): authenticate the evalboard upload with the existing storage credential The evalboard upload minted a GitHub OIDC token and exchanged it for an Azure credential, which needed a new Entra app registration, a federated credential scoped to a GitHub environment, and a container role assignment before it could work at all. The ADO pipelines already authenticate to this same storage account with `AZURE_STORAGE_KEY` / `AZURE_STORAGE_ACCOUNT`, and eval-runner passes that value straight to `BlobServiceClient`, so reusing it removes every one of those provisioning steps: two repo settings and nothing else. The Azure SDK sniffs the credential string, so the secret holds either the account key or a container-scoped SAS token with no change here. The SAS is preferable, since an account key is authority over the whole account including the nightly-history container, and this workflow runs its definition from whatever branch the dispatcher picks. Drops `id-token: write`, the `eval-upload` environment and the token-minting step. The gate moves to a job-env boolean because the `secrets` context is unavailable in any `if:`; deriving a boolean rather than exporting the secret keeps the credential in the upload step alone, matching how ADO scopes it. The container name is now hardcoded, since the unlisted evalboard source and the 14-day expiry rule both depend on ad-hoc runs never landing in `runs`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor(ci): pass the whole coder-eval command line through the action's args Follows the action's input simplification: it no longer promotes any of `coder-eval run`'s flags to named inputs, so the task globs and `--model` join the `-e`, `--type`, `-j`, `-v` and `-D` entries in the single `args` block the prep step already composed. The `model` step output goes away with them. Task globs are word-split when appended (TASK_GLOBS is a space-separated list from the partition job) but never pathname-expanded, which is an improvement on what the action used to do with them: the CLI expands them itself, so `**` works without `globstar` and a glob matching nothing exits 1 rather than reaching the CLI as a literal path. `step-summary: "false"` and `junit-path` are gone. The action writes no job summary at all now and reports `run-md-path` instead, so the append can happen after redaction, which is the order this workflow needs and previously had to work around. The append reads that output with the literal path as a fallback, because a composite action's outputs may not propagate from a step that exited non-zero and this step runs on failure. The JUnit report moves to /tmp/runs/junit.xml, derived from run-dir; nothing consumed it at the old path. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(ci): temporarily pin the action to the PR-147 head for an e2e dry run The `v0` tag still serves the ten-input action, so this workflow's `with:` block would have `args`, `extras`, `extra-packages` and `working-directory` silently dropped — GitHub ignores an input the referenced tag does not declare. Pinning the PR head validates the whole path before anything merges and before Release moves `v0`. REVERT TO @v0 BEFORE MERGING. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(ci): trim the comment bulk around the action split Comments only. Fifteen blocks cut to their contract, 118 lines out for 58 in, so the file sits at its existing 33% comment density instead of above it. Also drops a stale claim: the two-step-split banner said TASK_GLOBS "is handed straight to the action's `tasks` input, which word-splits it". There is no `tasks` input, and `args` entries are passed verbatim. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(ci): put the evalboard link at the top of the job summary The link was appended after run.md, which is roughly 40 lines of tables, so it landed below the fold and was easy to miss. Prepend rather than reorder the steps: the evalboard step moves /tmp/runs, which both the artifact upload and the verdict step read, so it has to stay last. $GITHUB_STEP_SUMMARY is an ordinary file, so the block is written to a temp file with the existing summary appended, then truncate-and-rewritten in place to keep the inode the runner tracks. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(ci): publish evalboard before the report so the link renders first The previous attempt prepended to $GITHUB_STEP_SUMMARY inside the evalboard step. That cannot work: the variable points at a SEPARATE file per step, and GitHub builds the job summary by concatenating them in step order, so the step read its own empty file and its block still rendered last. Verified on run 33564007553, where the link stayed at the bottom. Move the publish above the step that appends run.md, which is the only way to control position, and copy the run directory instead of moving it so the artifact upload and the verdict still find /tmp/runs. Also emit a ::notice::, which renders above the job list on the run page. Keep "Run Coder Eval" as the run-name prefix so the run list stays scannable by workflow as well as by glob, agent and actor. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(ci): grade the run directory, not its newest variant `--run-dir /tmp/runs` makes /tmp/runs the run dir itself, so its subdirectories are VARIANTS. The verdict step selected one with `ls -td /tmp/runs/*/ | head -n 1`, which is correct only while nightly.yaml declares a single variant: a second one would silently grade a single arm and could report a red run green. This is the same off-by-one the artifact globs in this branch already fix, one step lower. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * ci(coder-eval): consume the published action at @v0 The temporary SHA pin pointed at the PR branch of coder_eval#147, which is now merged and released as 0.11.6 with v0 promoted to it, so the eight-input surface this workflow passes is what v0 serves. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

fix(scorecard): count maturity carry-forward tasks in the Confluence scorecard (#2813) * fix(scorecard): count maturity carry-forward tasks in the Confluence scorecard The nightly runner skips a task that has passed 5 runs in a row and carries it forward as a synthetic SUCCESS row so the suite denominator stays at the full authored size. Those rows are written without a `variant_id`, while every genuinely-run row carries `variant_id: "default"`, so the scorecard's single-variant filter read them as a phantom `None` variant and dropped all of them: 545 of 1229 rows on the 2026-08-25 run, 856 of 1208 on 2026-08-21. This thinned every product row on the published card. Licensing showed 3/3 when 12 licensing tasks exist and 9 had been carried forward green, and the headline pass rate read 90.3% against evalboard's 97.2% for the same run. Keep carry-forward rows through the variant filter, derive the variant set from run rows only, and report the carried share per bucket so a row resting entirely on carried passes is visible as such. The measured-only rate is printed as a diagnostic and explicitly marked as not publishable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(scorecard): keep eval scores executed-only, count carried rows in pass/fail only Carried-forward rows all carry a uniform synthetic `weighted_score: 1.0`, which is not a measurement. Folding it into the means inflated the overall mean by +3.9pp on the 2026-08-21 run (94.5 to 98.4) and +24.6pp on 2026-08-25, which a reader comparing week over week would take for a quality improvement. Split the two columns by basis. Tests Pass/Fail and the Smoke / Integration+E2E tiers count carried rows as passes on the full known basis, which is the org card's own definition of the column and what evalboard reports. Every eval score and every mean now excludes them and stays executed-only. Every Build / Operate / Troubleshoot cell and every mean is byte-identical to what main produces today: 0 of 32 skills differ on the 2026-08-21 run, 0 of 35 on 2026-08-24. Only Tests Pass/Fail and the tier splits move. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

feat(tests): runnable docker setup for the preview Maestro SDK skills (#2728) * feat(tests): runnable docker setup for the preview Maestro SDK skills Running the three preview/uipath-maestro-{flow,case,bpmn} builder-SDK skills as the only skill catalog needed four non-obvious pieces of setup, none of them recorded anywhere. Each missing piece scores as a capability failure rather than a config error, so the results look plausible and are not. Dockerfile: add a runtime npmrc for the @uipath scope plus NPM_CONFIG_USERCONFIG. The build-time npmrc is written to globalconfig and deleted in the same RUN because it carries the literal token, so the image ships no npm auth at all; this layer adds a token-LESS npmrc referencing ${NODE_AUTH_TOKEN}, which npm expands at read time. NPM_CONFIG_USERCONFIG is what makes it reachable: npm resolves userconfig to $HOME/.npmrc, and the docker runner forwards --env HOME with the HOST value on purpose, overriding the image's /root, so npm looks in a directory the container does not have, never maps @uipath to GitHub Packages, and 404s on the public registry. Without this, `npm install @uipath/flow-sdk` fails and every builder-SDK compile fails with it. tests/experiments/preview-maestro-sdk.yaml: the catalog is narrowed by pointing agent.plugins[].path at preview/, which shadows the same-named v1 skills. That path does double duty as a bind mount, so narrowing it also drops the repo-root mount that nightly gets for free, and criteria across the flow and case suites that shell out to tests/tasks/**/_shared/*.py then exit 2. extra_mounts restores the repo root explicitly. File-based auth is documented as a commented block: its destination has to equal the host $HOME for the same forwarded-HOME reason, and mounting to /root/.uipath instead yields "Not logged in" on every tenant call. tests/docker/preflight.sh: one container, run with the host HOME forwarded exactly as the harness does, asserting the eleven preconditions a full run depends on -- login state readable, SDK installs, and check/compile/check plus the product scaffold all succeeding. A manual `docker run` without --env HOME authenticates against the image's /root and reproduces none of these failures, which is why they are easy to miss. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(tests): unify auth on the nightly mount, rename to flow-v2-* The preview experiment shipped its login mount commented out with a `$HOME/.uipath` destination, blocked on destination-side `$VAR` expansion (UiPath/coder_eval#128). It doesn't need it: nightly.yaml and smoke.yaml already authenticate the in-container `uip` from `~/.uipath:/.uipath:rw`, a literal destination that validates on the pinned coder_eval 0.10.2. Adopt that mount verbatim rather than deriving a "more correct" one. It is the arrangement with a 500-task/night track record, and a login the CLI cannot see fails tasks on their tenant calls, which scores as a capability problem rather than a config error. flow-v2-preflight.sh mounts the same destination, so its "uip reports a live login" check is now the empirical test of the unified path. same-ground-headtohead.yaml gets the same treatment, dropping both `/home/tmatup/...` hardcodes and the comment deferring them to UiPath/coder_eval#100. One `$VAR` destination survives, on the repo-root mount, and it is unrelated to auth: DockerDriverConfig can forward host env vars but cannot set container ones, so criteria see `$SKILLS_REPO_PATH` with its forwarded host value and the mount has to land there. That single line is what still wants coder_eval#128. Renames make the Flow v2 scope legible: tests/experiments/preview-maestro-sdk.yaml -> flow-v2-preview.yaml tests/docker/preflight.sh -> flow-v2-preflight.sh same-ground-headtohead.yaml keeps its name: per tests/README.md it is a general campaign comparison arm (pinned skills-image:sg1, SG_EMPTY_SKILLS blanking installed skills), not the v1 half of a Flow v2 head-to-head. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(tests): trim flow-v2 comments and drop product checks from preflight Comment volume was out of proportion to the config it explains, in the experiment, same-ground, the Dockerfile npmrc layer and the README. flow-v2-preflight.sh loses its six product-behaviour checks (flow check --source / compile / emitted / check --compiled / solution init / scaffold) and the inline TS flow they needed. Those are what the eval measures, and pinning CLI verbs in a gate script only rots. What remains is the six config assertions whose failures score as capability problems instead: forwarded HOME, npm userconfig, @uipath registry, token reachability, a live uip login, and an in-sandbox flow-sdk install. 82 lines to 52. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(tests): drop flow-v2-preflight.sh for a one-line login check The script's only load-bearing assertion was that the `/.uipath` login mount resolves under the forwarded host HOME. That is one `docker run`, now in tests/README.md. The rest either duplicated what a failing run already reports or pinned CLI verbs that drift. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(tests): move preview npm auth out of the shared image into pre_run The npmrc layer set ENV NPM_CONFIG_USERCONFIG on tests/docker/Dockerfile, which changes npm resolution for every suite, not just the Flow v2 preview. Several skills tell agents to `npm install -g @uipath/cli`, which is public-npm-only for stable releases and carries no registry override, so a global @uipath -> GitHub Packages mapping is a hazard the preview run has no business creating. Revert the Dockerfile and write the npmrc from the experiment's pre_run instead, into $HOME, which is where npm resolves userconfig, so it holds whatever HOME the runner forwards and needs no NPM_CONFIG_USERCONFIG at all. pre_run executes inside the sandbox before the agent and fails the task loudly on error, so a broken write cannot masquerade as a capability miss. ${NODE_AUTH_TOKEN} stays literal; npm expands it at read time. Only flow-v2-preview.yaml needs this. No other experiment installs an @uipath package in-sandbox. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ci(rpa-smoke): cap anthropic below 1.0 for the pinned coder-eval anthropic 1.0.0 (2026-08-20) moved its HTTP layer to httpx2. coder-eval 0.10.2 imports httpx in judge_bedrock without declaring it, so it only ever resolved transitively through anthropic 0.x. On a fresh install today llm_judge fails to import and every task carrying an llm_judge criterion errors at setup with score 0.00. Reproduced against the released wheel, nothing branch-specific: uv pip install "coder-eval==0.10.2" -> anthropic 1.0.0, no httpx -> import coder_eval.criteria.llm_judge ModuleNotFoundError: No module named 'httpx' uv pip install "coder-eval==0.10.2" "anthropic<1.0" -> anthropic 0.125.0, httpx 0.28.1, llm_judge imports coder_eval main already switched to httpx2, so this cap comes off with the next .coder-eval-version bump. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * revert: "ci(rpa-smoke): cap anthropic below 1.0 for the pinned coder-eval" This reverts commit 93f285037. The forward fix is already open as #2707, which moves the pin to 0.11.1 — the coder_eval release carrying beeceddc, "bump anthropic to 1.0.0, migrate Bedrock judge path to httpx2" — and exempts anthropic from the safe-chain package-age gate in the same four workflows. Keeping the cap would break that merge rather than help it: 0.11.1 declares anthropic>=1.0.0,<2.0.0, so the two constraints are unsatisfiable and the install step would fail outright. uv pip install "coder-eval==0.11.1" "anthropic<1.0" -> No solution found when resolving dependencies The cap also would not have turned the check green. With httpx restored the tasks reached the agent and then hit a second, unrelated Windows break: claude-agent-sdk 0.2.144 cannot find a native claude.exe, so all three crash with agent_crash after 3 attempts. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore(tests): bump the coder-eval pin to 0.11.2 UiPath/coder_eval#128 shipped in 0.11.2, so `_validate_extra_mount` now expands `~` and `$VAR` in a destination and the flow-v2-preview repo-root mount loads instead of aborting. Carries 0.11.0's breaking changes into the whole suite: the cap-drop anti-cheat window on every `docker run`, and directory-only `task.reference`. The ~298 `reference: {file:}` tasks under uipath-troubleshoot are skipped with a migration error until #2707 lands their migration. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ci(smoke): exclude claude-agent-sdk 0.2.144 on the Windows RPA runner 0.2.144 is the first release in months published without a win_amd64 wheel, so uv falls back to the sdist and nothing bundles claude.exe. All three RPA smoke tasks crash at agent_crash before doing any work. coder-eval requires claude-agent-sdk>=0.2.124 with no upper bound, so exclude that single release rather than capping the range: 0.2.143 has the wheel, and a later release that restores it resolves normally. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

ci: move Claude jobs to Bedrock and drop unused secret references (#4146) * ci: move Claude jobs to Bedrock and drop unused secret references ANTHROPIC_API_KEY and UV_INDEX_UIPATH_PASSWORD were revoked and are not being replaced. - Claude PR Review and the task lint now run claude-code-action on Bedrock (use_bedrock + AWS_BEARER_TOKEN_BEDROCK, us.anthropic.claude-opus-4-6-v1, same model as before). - The coder-eval steps no longer forward ANTHROPIC_API_KEY: they already run on the Bedrock backend, judges go through the litellm route, and the LLM reviewer the comment described no longer exists in coder-eval. - The UiPath-Internal NuGet feed allows anonymous reads, so it is added without credentials. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci: run Claude PR review and task lint on Opus 5.5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>