Akshaya Shanbhogue

test: remove $TASK_DIR/$SKILLS_REPO_PATH host-path deps from task yamls (#3271) * test: remove $TASK_DIR/$SKILLS_REPO_PATH host-path deps from task yamls Task sandboxes (docker/tempdir drivers) don't have access to a full host repo checkout, so scripts/fixtures a task needs must be staged via sandbox.template_sources (visible to the agent, used for pre_run/post_run tooling) or reached via $REFERENCE_DIR (mirrors reference.directory on disk, used for grading/run_command/llm_judge criteria, invisible to the agent). - Migrated uipath-admin, uipath-governance (incl. compliance-pack), and uipath-review (agents/* and rpa/*) task families off $SKILLS_REPO_PATH: moved shared helper scripts from _shared/ to _setup/ where they were actually pre_run/post_run tooling, added sandbox.template_sources entries to stage them into the sandbox, and switched run_command criteria to the $REFERENCE_DIR-relative runpy bootstrap. - Fixed the uipath-solution deploy/config-lifecycle e2e tasks and pack_determinism_integration.yaml to stage their zip/solution fixtures via template_sources instead of an absolute $SKILLS_REPO_PATH path. - Extracted a standalone flow_project_finder.py (verbatim copy of _shared/flow_check.find_project_dir, no grading logic) so the ixp handoff grader's pre_run/post_run tooling no longer needs $SKILLS_REPO_PATH to reach a shared grading module. - Cleaned up assorted stale $SKILLS_REPO_PATH/$TASK_DIR references in docstrings and comments across uipath-maestro-flow, uipath-maestro-bpmn, uipath-insights, and elsewhere. - Added include_reference: false to llm_judge success criteria that were missing it. - Left a documented set of exceptions in place (optional ambient skills/<name>/references/*.md strengthening checks that degrade gracefully, tasks needing genuinely broader host-repo access, and meta-tests recognizing the token as legacy text). - Repointed several checker unit tests at scripts that had moved to _shared/_setup during this migration (cleanup_solutions.py, check_athena_cm_event_case.py, check_athena_cm_event_rules.py, check_cm_golden_case.py, check_sla_from_sdd.py, check_timer_connector.py, seed.py) — these were breaking the maestro-case and maestro-flow checker unit test CI jobs. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(tests): close admin/governance checker path bugs from host-path removal Two issues surfaced after removing $SKILLS_REPO_PATH/$TASK_DIR from the admin/governance task yamls: - reference.directory: . only copied the leaf task folder into $REFERENCE_DIR, so verify_*.py scripts invoked via runpy.run_path() couldn't find admin_helpers.py/gov_helpers.py in the family-level _setup/. Widened to .. and prefixed the $REFERENCE_DIR script path with the leaf folder name across all 54 affected yaml files (identity, authz, apms, audit, access-policy, aops-policy). - audit_helpers.py — pure grading logic documenting the audit checker's record-schema/casing rules — lived in uipath-admin/_setup/, which is wholesale-mounted via template_sources into 36 unrelated identity/authz/apms agent sandboxes. Moved it to audit/, colocated with its three consumers, closing that leak. admin_helpers.py and gov_helpers.py stay in _setup/ since they're generic CLI plumbing with no judging-specific logic. Also restored explicit sys.path.insert in the three verify_audit_*.py scripts and in the relocated audit_helpers.py — runpy.run_path() does not auto-add a script's directory to sys.path the way direct `python3 script.py` execution does, so colocation alone isn't sufficient. Verified by reproducing the grading invocation in isolation for identity/audit/governance cases (reaches real logic instead of ModuleNotFoundError) and by running pytest tasks/uipath-admin/ (33 passed). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(tests): address PR #3271 review findings — staging bugs, answer-key leaks, doc drift Full remediation of the automated 8-axis review on #3271 (3 critical + 5 high blockers, plus non-blocking/nit findings), each independently verified before fixing: Blockers: - 17 uipath-review/agents tasks staged `path: ../_setup`, one level short of the real `_setup/` — fixed to `../../_setup`. - Fixing that path re-activated a real answer-key leak: 16 `_setup/inject_*.py` scripts named the exact rule_id their own grader greps for, once staged into the agent's own sandbox. Reworded every literal rule-id token out of those docstrings/comments. The 17th task (stale_generated_files) failed agents in the opposite direction — its checker matched a canary literal anywhere in the report, including the sandbox's own harness scaffolding; scoped the assertion to defect-section citations only. - 5 uipath-ixp tasks staged `../_shared/fixtures` with no mount_point (defaults to sandbox root, but prompts say `./fixtures/`) — repointed to the already-migrated `../_setup/fixtures` with `mount_point: fixtures`. - `jira_is.py` was never relocated for 5 maestro-flow Jira tasks: pre_run's `_setup/seed_jira.py`/`teardown_jira.py` bare-import it, but it still sat at the task root (never staged) — moved into each task's own `_setup/`. Grading's `_shared/check_jira_*.py` also import it locally — added a merged `_shared/jira_is.py` superset (FOLDER_NAME/issue_absent from the escalation copy, myself()/retry plumbing from the create-issue copy). - `uipath-governance/cleanup_policy.py` was referenced from `compliance-pack/_setup/cleanup_policy.py` but never moved there — moved. - `seed_coded_order.py` kept `parents[3]` after being staged into the sandbox; the branch silently flipped to a fallback that calls a nonexistent `df choice-sets add-value` verb — pointed at the co-staged sibling `seed_choice_set.py` instead, removed the broken fallback. - Split the fused 1057-line `ixp/e2e_03.../_setup/handoff.py`: agent-visible `handoff.py`/`handoff_tenant.py` keep only pre_run/post_run tooling and tenant plumbing; the primary grading gate (`check_main` and its answer-key helpers) moved to `_shared/check_ixp_handoff.py`, reached only via $REFERENCE_DIR. Deleted the now-unused `_setup/flow_project_finder.py` copy. Repointed test_handoff.py and e2e_04's criterion accordingly. - 4 checkers (`check_cm_golden_{seed,bindings,semantics}.py`, `check_is_smoke.py`) resolved a `fixtures/sdd.md` / fixture path that never moved with them — pointed at the reference-mirror location instead of the agent-writable sandbox copy. Non-blocking / nits / what's-missing, also addressed: - `test_cleanup_codedapp_folder.py` repointed to the moved `_setup/` script, converted to real pytest functions, added `pytest-coded-apps-check` and `pytest-review-check` CI jobs (test-helpers.yml). - Dead `sys.path.insert` entries removed from 3 api-workflow checkers. - `coded_in_flow_e2e.yaml`'s `reference.directory` widened to `../..` to match its siblings and actually reach `_shared/`. - `smoke-skills.yml`'s `resolve_task_yaml()` now falls back to a skill's full smoke suite when a changed `_shared/`/`_setup/` file resolves to no owning task yaml (previously silently selected nothing). - CODEOWNERS: restored specific ownership for 10 checker files relocated into `_shared/` during the migration (guardrails + maestro-flow connector checkers), so they don't silently fall back to the broader family default. - Orphaned YAML comments reattached to the settings they describe (ixp/name_resolution.yaml, e2e_04_build_mechanics.yaml). - Stale doc drift fixed: tests/README.md (`_setup` vs `_shared` shape + the old $SKILLS_REPO_PATH example), uipath-maestro-bpmn/README.md, generate-task.md scaffolder instructions, test-task-template.yaml, flow-v2-preview.yaml's stale grader rationale, stage_shared.sh's still-load-bearing scope, and the troubleshoot smoke task's now-outdated "coder-eval doesn't inject TASK_DIR" comment (documented as the intentional broad-host-access exception it always was). Verified: pytest green per-family (maestro-flow 1235, maestro-case 139, agents 17, admin 33, review 41, coded-apps 6, maestro-bpmn 19, planner 89), all changed YAMLs parse, scripts/check-task-driver.py clean, and every relocated/rewritten script's import path traced by hand against its actual staged or reference-mirrored location. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(ci): register review/coded-apps checker jobs as required checks test-helpers.yml added pytest-review-check and pytest-coded-apps-check in the previous commit but docs/REQUIRED-CHECKS.md's target set table was never updated to match, tripping test_every_test_helpers_job_is_required. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(tests): remove $SKILLS_REPO_PATH host-path dep from disabled eval_run task Missed in the earlier sweep since .disabled files aren't discovered by coder_eval. Stages _setup via sandbox.template_sources like its active sibling tasks (e.g. single_node/switch), instead of reaching cleanup_solutions.py through the host repo path. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

test: gate new $TASK_DIR/$SKILLS_REPO_PATH task host-path references (#3356) * test: gate new $TASK_DIR/$SKILLS_REPO_PATH task host-path references #3271 migrated the task corpus off these two retired host-path escape hatches, but nothing stopped later commits from reintroducing them — 3 files had drifted back since. Add a CI gate that diffs new task-yaml lines against the PR base and fails on any new occurrence (a one-way ratchet, matching check-skills-sh.py's --baseline-ref pattern; it leaves #3271's documented pre-existing exceptions alone). Clean up the 3 drifted files to pass it: - uipath-maestro-bpmn customer_escalation_triage: move seed/teardown/ escalation_is scripts into a task-local _setup/, move the family's cleanup_solutions.py from _shared/ to a family-level _setup/ (pre_run/ post_run tooling, not grading), and reach the checker scripts via reference.directory + $REFERENCE_DIR. - uipath-maestro-flow datafabric_native_contract_intake_pipeline: same pattern, staging its cross-task ensure_entity.py dependency from the sibling connector_features/datafabric_connector/_setup/ alongside the family _setup/. - Ported the $REFERENCE_DIR resolution support the maestro-flow _shared/test_criterion_budgets.py already had into the bpmn family's copy, so its debug-timeout budget guard keeps pricing the migrated criterion instead of silently dropping it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * chore(tests): install harbor and codex extras alongside litellm The coder-eval install spec only pulled the litellm extra; local setup for harbor- and codex-backed runs required manually reinstalling with broader extras. Install all three by default. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

feat(tests): route checker LLM through litellm gpt-5.6-luna via coder-eval 0.11.5 (#2864) * feat(tests): route checker LLM through litellm gpt-5.6-luna via coder-eval 0.11.3 Bump coder-eval to 0.11.3 and set checker_context.api_route defaults (default/smoke/smoke-windows/nightly/activation experiments) so the success-checker/judge calls litellm -> gpt-5.6-luna instead of the built-in default, resolving credentials from CODEX_BASE_URL/CODEX_API_KEY. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * chore(tests): bump coder-eval pin to 0.11.4 0.11.4 bakes --extra litellm into the coder-eval-agent base image (UiPath/coder_eval#142), which tests/experiments/*.yaml's litellm-routed checker_context needs available in-container under driver: docker. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(tests): scope checker_context.api_route to per-task, not experiment defaults Addresses PR #2864 review. coder-eval rejects route: litellm outright on any task with simulation.enabled or an enabled agent_judge criterion (the simulator/agent_judge run as real Claude Code subprocesses, incompatible with an arbitrary litellm-fronted gateway) -- see orchestrator._reject_litellm_eval_route_if_unsupported. 303 of 359 llm_judge tasks in this repo have simulation.enabled, so the previous experiment-level default broke all of them at setup, including one smoke-tagged task (uipath-maestro-flow/interactive/solution_select.yaml). - Remove checker_context from experiments/{default,smoke,smoke-windows, nightly,activation}.yaml. activation.yaml has zero llm_judge criteria, so it gets no replacement. - Add checker_context.api_route directly to the 56 task YAMLs that have an llm_judge criterion and no simulation.enabled -- the only tasks it's safe and meaningful for. This also fixes same-ground-headtohead.yaml and flow-v2-preview.yaml, which never got the experiment-level block: a task-level setting applies under any experiment that runs the task. - Fix model: azure/gpt-5.6-luna, not the bare id -- litellm resolves an unprefixed model to provider `openai`, which would silently ignore the Azure-shaped `params.api_version`. Matches coder_eval's own TASK_DEFINITION_GUIDE.md example verbatim. - tests/Makefile: install the litellm extra for local `make install`, and point at the new README section for CODEX_BASE_URL/CODEX_API_KEY. - tests/README.md: document checker_context.api_route, the litellm/ simulation incompatibility, and what each driver needs (host extra for tempdir, baked into the agent image for docker -- coder-eval-agent bags it as of 0.11.4). - smoke-skills.yml: note the one deliberate exception to "the grader stays on Claude" now that a few tasks route their own llm_judge through the gate's CODEX_* endpoint. Windows RPA smoke (smoke-rpa-skills.yml) and skill-activation-gate.yml need no change: no task either runs carries checker_context now that it's task-scoped. run-coder-eval.yml's docker/Windows jobs already installed the litellm extra and passed CODEX_BASE_URL/CODEX_API_KEY through. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * revert(tests): restore experiment-level checker_context.api_route default Reverts e36dce6ff. The per-task scoping worked around coder_eval rejecting route: litellm on any task with simulation.enabled or an enabled agent_judge criterion -- but the real fix belongs upstream. coder_eval now resolves the simulator's route independently of checker_context.api_route entirely (Orchestrator.simulator_route), so route: litellm is safe as an experiment default even with simulation.enabled: true. - Restore checker_context.api_route to defaults/{default,smoke, smoke-windows,nightly,activation}.yaml, keeping the azure/gpt-5.6-luna model fix (bare gpt-5.6-luna resolves to provider openai, which silently ignores the Azure-shaped params.api_version). - Remove the 56 per-task checker_context blocks -- redundant now that the experiment default covers every llm_judge task safely. - tests/Makefile: keep the [litellm] extra for local `make install` (still needed for driver: tempdir now that this is the default, not a per-task exception). - tests/README.md: rewrite Checker Context to describe the new simulator-decoupled default instead of "set this per-task, never as an experiment default." Still calls out that agent_judge + route: litellm remains unsupported (no agent_judge criteria in this repo today). - smoke-skills.yml: drop the per-task-exception comment on AGENT_MODEL, no longer accurate now that ~all llm_judge tasks route through litellm by default, not "a handful." NOT bumping tests/.coder-eval-version yet -- the coder_eval fix (simulator_route decoupling) is committed on akshaya/scope-litellm-route-to-llm-judge but not released. Merging this against the current 0.11.4 pin would reintroduce the original CI failure (route: litellm rejected outright on simulation-enabled tasks). Bump the pin once coder_eval ships a release containing that fix. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

chore: update coder-eval version (#2504) * chore: update coder-eval version Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(activation): migrate stop_when to stop_early blocks for coder-eval 0.9.5 coder-eval 0.9.5 removed the stop_when field (breaking change in #78); per-criterion stop_early blocks replace it. stop_when: auto -> stop_early: {on_pass: stop}, the equivalent arming for dataset-fanned skill_triggered rows. Unblocks the dry-run gate, which reds when activation runs 0 cases. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(activation): drop removed run_limits.stop_early master arm for coder-eval 0.9.5 0.9.5 rejects run_limits.stop_early: true at CLI resolution (arming is per-criterion via stop_early blocks, already on every skill_triggered criterion); false survives only as the run-level kill switch. The invalid key crashed the activation suite before it wrote run.json, leaving activation_cases=0 and reding the dry-run gate. Also refresh the stale stop_when wording in the experiment comments and README. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

chore: bump coder-eval version to 0.12.1 (#3269) Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

ci: install coder-eval from public PyPI (#1969) Drop the UV_EXTRA_INDEX_URL private Azure DevOps feed overrides from every coder-eval install step. coder-eval is now published to public PyPI, so uv pip install resolves it (and its deps) from the default index. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

chore(tests): bump coder-eval to 0.12.4 (#3410) 0.12.4 ships the run-layout change that splits the run directory into two independent, late-resolved directory templates (--logging-dir for task.json/task.log/task.html/trajectory.json, --artifacts-dir for the agent's workspace). Under Harbor this removes the copy step and the duplicated workspace that appeared when artifacts were structurally a child of the run directory. Validated on this repo before bumping: the 10 sampled Harbor tasks reconvert cleanly (10/10 export, byte-identical to the pre-release build apart from their own paths) and rerun green, with the agent logs directory holding exactly setup/task.execute.json/task.html/task.json/task.log/trajectory.json, /work collected once, no duplicated workspace, and no run-level files leaking into the logs directory. A docker-driver run was checked separately and still reproduces the historical <variant>/<task>/<repeat>/artifacts/<task> layout. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

test: add num_retries to checker_context api_route across experiments (#3009) Ensures the litellm judge route retries on transient failures instead of failing a row outright.

chore: update coder-eval version (#2337)