AWS
sagemaker-python-sdk — Engineering Performance
7 engineers all time · Jan 2025 – Sep 2026 · built 2026-09-10 · GitHub
Performance snapshot
Today's rolling 90-day reading for sagemaker-python-sdk, compared with the start of the series. Pick a window to move that comparison point.
Avg. perf / dev / mo
+12571.4%
0.02 → 2.11 ETV
Active engineers
+250.0%
2.0 → 7.0
Features
+3.7pp
20.0% → 23.7%
vs. AWS
2.5x
0.05x → 2.5x · +154% above
sagemaker-python-sdk vs. AWS
Per-engineer ETV for sagemaker-python-sdk against AWS as a whole. Both lines are 90-day rolling averages scaled to a 30-day month, so they share one axis and can be read against each other at any point. Pick a window to zoom the chart to it.
sagemaker-python-sdk
2.11ETV / dev / mo
+1.15 (+119.8%) past 90d
AWS
0.83ETV / dev / mo
+0.33 (+66.0%) past 90d
Performance Composition
Each month's output split by type of work: Features (new value), Maintenance (sustaining systems), Tests, Docs, and Fixes (rework). The yellow line is output per engineer, so when it rises each engineer is delivering more, whatever the team size did. Unit: Engineering Throughput Value (ETV).
Engineering capacity
Effective engineers behind sagemaker-python-sdk, against its pre-AI baseline. Each subject has its own: sagemaker-python-sdk's is 0.19 ETV / dev / mo, its first reading in Q2 2025. Per-engineer ETV divided by that gives a capacity multiple, and that multiple applied to the engineers active in the trailing 90 days turns it into engineer-equivalents. The line is the real headcount, so the gap between line and area is what the leverage is worth. Because each baseline is its own, every subject opens at 1.0x on its first day: multiples measure improvement and are not comparable between subjects.
Effective engineers
78engineer-equivalents
+71 engineers above real headcount
Real engineers
7engineers
active in the trailing 90 days
Capacity vs pre-AI
11.1x
per engineer, vs 0.19 pre-AI
Knowledge concentration
How dependent is this repo on a small number of engineers? Higher top-1 share = higher key-person risk.
lucasjia-aws owns 21.6 % of commits.
Behind the numbers
Written summary of the work completed each month.
No monthly reports available yet.
Top engineers
Most impactful commits
Top 10 by ETV in the all-time window.
- 24.4ETVNova release (#5969) * Add master-mtrl-trainer branch to PR checks * Add master-mtrl-release branch to PR checks * Add new branch for nova reconciliation in PR checks * wip: RLVRTrainer lambda arn support (#2025) * wip: RLVRTrainer lambda arn support * fix: add duplicate check in evaluator creation + added tests * fix: unit test bugs * feat: add optional hub_name param to AI registry class * fix bugs in new rlvr integ tests * fix: improve exception handling in finetune utils * fix: logging in _extract_evaluator_arn * fix: revert extra hub_name param in air_hub * fix: add gpu intensive marker for new integ tests * doc: add section in example notebook for passing lambda arn to RLVR Trainer --------- Co-authored-by: Syed Jafri <syedjfr@amazon.com> * feat: add is_multimodal utils function (#2033) * feat: add is_multimodal utils function * fix: Make method name Nova specific: nova_dataset_is_multimodal * fix: move to is_multimodal to data_utils file * fix: rename to is_multimodal_data * chore: ran code formatter * doc: make docstring nova agnostic * feat: type: Dataset support for is_multimodal method * feat: add is_multimodal param to model customization trainers * observability: add log statement for is_multimodal_data --------- Co-authored-by: Syed Jafri <syedjfr@amazon.com> * feat(train): Add 3-level recipe override support with get_resolved_recipe() (#2034) * Update pr-checks-master.yml * feat(train): Add 3-level recipe override support with get_resolved_recipe() Adds recipe file + programmatic overrides to trainers (SFT, RLVR, DPO), evaluators (BenchMarkEvaluator), and ModelTrainer. Users can now pass a YAML recipe file and/or overrides dict, then inspect the fully merged configuration via get_resolved_recipe() before submitting a job. Key additions: - RecipeResolver class: handles Hub template rendering, OmegaConf 3-way merge, validation against override-params spec, and caching - recipe/overrides params on SFTTrainer, RLVRTrainer, DPOTrainer, BenchMarkEvaluator constructors - get_resolved_recipe() on all trainers, evaluators, and ModelTrainer - 40 unit tests covering resolver logic and trainer integration - Example notebooks for SFT, evaluator, and ModelTrainer workflows * refactor: Address PE review - extract mixin, fix mutation bug, remove SSRF vector - Move get_resolved_recipe() to BaseTrainer (eliminates copy-paste across SFT/RLVR/DPO trainers) - Deep-copy user_dict before stripping protected keys (fixes mutation bug) - Remove HTTP/HTTPS URL support from recipe loading (SSRF prevention) - Clarify overrides docstring: introspection-only, not plumbed to train() - Remove unused import (from logging import exception) * feat(train): Add integration tests for recipe override, fix enum validation - Add integ tests for SFTTrainer and BenchMarkEvaluator get_resolved_recipe() (tests run against live Hub without submitting training jobs) - Fix enum validation to skip empty-string defaults (Hub specs sometimes have empty-string defaults that don't match their own enum constraints) - All 45 tests pass (30 unit + 10 trainer + 5 integ) * refactor: Move recipe support to BaseEvaluator, fix docstrings - Move recipe/overrides fields and get_resolved_recipe() from BenchMarkEvaluator to BaseEvaluator (now available to CustomScorer, LLMAsJudge, and all future evaluators) - Fix docstring: evaluators do 2-level merge (recipe < overrides), not 3-level (no Hub template for eval) - Fix ModelTrainer docstring: get_resolved_recipe() works for all recipe types, not just Nova - Remove duplicate get_resolved_recipe() from BenchMarkEvaluator * feat: Wire recipe into train()/evaluate() + add unit tests - SFT/RLVR/DPO train(): resolved recipe values now override matching keys in final_hyperparameters before job submission - Evaluators: _get_effective_hyperparameters() returns resolved recipe values when recipe/overrides are provided, used by evaluate() - Add 4 unit tests verifying recipe values flow into train()/evaluate(): - SFT train() applies recipe overrides to hyperparameters - SFT train() without recipe uses hyperparameters unchanged - Evaluator _get_effective_hyperparameters with recipe - Evaluator _get_effective_hyperparameters without recipe (fallback) - Fix comment: "overrides > recipe > Hub defaults" - Move recipe/overrides fields to BaseEvaluator for all evaluators * test: Add ModelTrainer integ tests for get_resolved_recipe() - test_model_trainer_get_resolved_recipe_with_local_yaml: verifies overrides win over recipe file values, recipe values preserved - test_model_trainer_get_resolved_recipe_overrides_only: verifies recipe-only path (no overrides) returns file values as-is - test_model_trainer_get_resolved_recipe_is_idempotent: verifies caching, deep-copy isolation from mutations * refactor: Extract flatten_resolved_recipe utility, DRY up trainers/evaluators - Add flatten_resolved_recipe() to recipe_resolver.py as shared utility - BaseTrainer._apply_recipe_to_hyperparameters() uses it (called by SFT/RLVR/DPO train()) - BaseEvaluator._get_effective_hyperparameters() uses it (called by BenchMark/CustomScorer evaluate()) - Eliminates duplicated flattening logic across all surfaces * feat: Add recipe/overrides support to MultiTurnRLTrainer and MultiTurnRLEvaluator - MultiTurnRLTrainer: add recipe/overrides params, wire into train() via _apply_recipe_to_hyperparameters() - MultiTurnRLEvaluator: uses _get_effective_hyperparameters() from BaseEvaluator (recipe values flow into template context) - Add 4 unit tests for MTRL trainer and evaluator recipe integration - Total: 56 tests passing * feat: Enable full recipe template override for non-spec keys Allow power users to override any key in the Hub recipe template, not just the UI-exposed spec subset. Fetches SmtjRecipeTemplateS3Uri at init time and uses it as the base layer for recipe resolution. - Add full_recipe_template param to RecipeResolver and resolve_recipe() - Fetch full recipe YAML in _get_fine_tuning_options_and_model_arn() - Remove filter in _apply_recipe_to_hyperparameters (all scalar keys flow through) - Skip nested dict/list values during hyperparameter flattening - Move in-function imports to file level - Add unit tests (TestFullRecipeTemplate, TestBuildKeyPathMap) - Add integration tests (TestSFTTrainerFullRecipeOverrideInteg) - Update notebook with power-user override example * Add branch for master-nova-reconciliation to workflow * fix: Deep-flatten nested recipe keys into hyperparameters flatten_resolved_recipe now recursively walks nested dicts to extract all scalar leaf values by key name. This fixes the issue where nested recipe fields like lr_scheduler.warmup_steps were dropped during _apply_recipe_to_hyperparameters because they were dict values. Users can now override nested keys: overrides={"training_config": {"lr_scheduler": {"warmup_steps": 30}}} and they correctly flow through as flat hyperparameters to the training job. * test: Add unit and integ tests for nested recipe key override flattening - Unit: test_deeply_nested_peft_keys_flow_into_hyperparameters - Unit: test_no_dicts_or_lists_in_final_hyperparameters - Integ: TestSFTTrainerNestedRecipeOverrideInteg (3 tests) - test_sft_nested_override_flows_to_hyperparameters - test_sft_nested_defaults_preserved_in_hyperparameters - test_sft_recipe_file_overrides_nested_keys * docs: Add nested recipe attribute example to notebook Demonstrates how deeply nested recipe keys (lr_scheduler.warmup_steps, peft.rank) are handled: overridden via recipe_overrides and auto-flattened into scalar hyperparameters for the training API. * feat: Add Nova SMI config bounds validation to ModelBuilder (#2040) * feat: Add Nova SMI config bounds validation to ModelBuilder Validates user-provided CONTEXT_LENGTH and MAX_CONCURRENCY against supported Nova inference tier limits before deployment. Previously, invalid configurations would only fail at container runtime with cryptic errors. Changes: - Add _NOVA_SMI_TIERS table with per-(model, instance) tier bounds - Add _validate_nova_smi_config() that checks env vars against tiers - Fix env var merge order so user overrides take priority over defaults - Wire validation into both Nova deploy paths - Add unit tests covering valid configs, boundary violations, and skip cases * refactor: Use _NOVA_HOSTING_CONFIGS for SMI validation instead of separate tier map Remove the duplicate _NOVA_SMI_TIERS map and rewrite _validate_nova_smi_config() to validate against the existing _NOVA_HOSTING_CONFIGS. This avoids maintaining two separate maps that must stay in sync. * refactor: Add Tiers to _NOVA_HOSTING_CONFIGS for accurate per-tier validation Embed full (max_context_length, max_concurrency) tier bounds directly into each _NOVA_HOSTING_CONFIGS entry. This removes the need for a separate _NOVA_SMI_TIERS map while preserving accurate intermediate-tier validation (e.g., at 64k context on p5, max concurrency is 32 not 8). * docs: Add env_vars configuration example for Nova deployment Add a section to the deployment notebook showing how to set CONTEXT_LENGTH and MAX_CONCURRENCY when deploying Nova models, including the tier bounds table and an example of validation error output. * fix: Add missing (256000, 2) tier for NOVA_LITE_2 on ml.p5.48xlarge Sync with Forge SDK SUPPORTED_SMI_CONFIGS which includes a 256k context tier that was missing from our config. * Feat/inspect ai evaluator (#2039) * feat: Support InspectAI evaluation. * Add InspectAI evaluation demo notebook * address black and isort * address flake8 * fix: accept trainer as model param, make region us agnostic --------- Co-authored-by: Luke Luneau <luluneau@amazon.com> * Iam role creation (#2041) * feat: Add Nova SMI config bounds validation to ModelBuilder Validates user-provided CONTEXT_LENGTH and MAX_CONCURRENCY against supported Nova inference tier limits before deployment. Previously, invalid configurations would only fail at container runtime with cryptic errors. Changes: - Add _NOVA_SMI_TIERS table with per-(model, instance) tier bounds - Add _validate_nova_smi_config() that checks env vars against tiers - Fix env var merge order so user overrides take priority over defaults - Wire validation into both Nova deploy paths - Add unit tests covering valid configs, boundary violations, and skip cases * refactor: Use _NOVA_HOSTING_CONFIGS for SMI validation instead of separate tier map Remove the duplicate _NOVA_SMI_TIERS map and rewrite _validate_nova_smi_config() to validate against the existing _NOVA_HOSTING_CONFIGS. This avoids maintaining two separate maps that must stay in sync. * refactor: Add Tiers to _NOVA_HOSTING_CONFIGS for accurate per-tier validation Embed full (max_context_length, max_concurrency) tier bounds directly into each _NOVA_HOSTING_CONFIGS entry. This removes the need for a separate _NOVA_SMI_TIERS map while preserving accurate intermediate-tier validation (e.g., at 64k context on p5, max concurrency is 32 not 8). * docs: Add env_vars configuration example for Nova deployment Add a section to the deployment notebook showing how to set CONTEXT_LENGTH and MAX_CONCURRENCY when deploying Nova models, including the tier bounds table and an example of validation error output. * fix: Add missing (256000, 2) tier for NOVA_LITE_2 on ml.p5.48xlarge Sync with Forge SDK SUPPORTED_SMI_CONFIGS which includes a 256k context tier that was missing from our config. * docs: Add IAM role auto-creation interfaces notebook Demonstrates the resolve_or_create_role auto-resolution behavior across the trainer, evaluator, model builder, pipeline, and feature store entry points. * feat: Auto-create least-privilege IAM roles when none provided Adds resolve_or_create_role() in sagemaker-core that resolves an existing role (explicit, caller identity, or default auto-role) via permission simulation and only creates a new least-privilege role when none is sufficient. Idempotent get-or-create with policy config in iam_policies.json (shipped as package-data). Wires it into the training, evaluation, serving, pipeline, and feature store entry points, replacing get_execution_role() fallbacks and the pipeline 'IAM role is required' ValueError. Updates affected unit tests and adds resolver, serving, and feature_store coverage. Known follow-up: feature_scheduler.put_trigger() still uses get_execution_role (EventBridge Rule needs events.amazonaws.com trust). * refactor: inline IAM policy config as Python constant; relocate example notebook - Convert iam_policies.json to a Python constant (IAM_POLICY_CONFIG in iam_policies.py). Source modules are always packaged, so this removes the need for the helper/*.json packaging references that would otherwise be required for the resolver to find the config at runtime in installed wheels. - Drop the now-unnecessary 'recursive-include src/sagemaker/core/helper *.json' (MANIFEST.in) and 'helper/*.json' package-data (pyproject.toml). - Resolver imports the constant instead of reading the bundled JSON via importlib.resources. - Move notebooks/iam_role_autocreate_interfaces_real.ipynb to v3-examples/iam_role_autocreation.ipynb. * refactor: Hoist IAM resolver imports to module top, document role resolution order Moves the resolve_or_create_role imports from inline (function-body) to the top of each consuming module. Adds comments at each call site documenting the resolution order: explicit role > caller session role (if sufficient) > on-demand least-privilege role. Updates test patch targets to patch resolve_or_create_role in each consumer's namespace (where it is now looked up) rather than the source module. * test: Mock resolve_or_create_role in train/dataset tests after IAM auto-create The IAM role auto-creation change (23f8c9b4) rewired TrainDefaults.get_role() to call resolve_or_create_role(), which issues boto STS calls and subscripts the responses. Tests that build a ModelTrainer/DataSet without an explicit role now hit that path with spec'd/bare mocks and fail. Patch resolve_or_create_role (matching the test_defaults.py pattern): - model_trainer: patch in the shared autouse modules_session fixture - dataset: add decorator to the two S3-location create tests * feat(iam): address PR review on role auto-creation Addresses Vaishnavi's review comments on PR #2041: - Policy drift: _ensure_policies_attached now syncs an existing policy's document via CreatePolicyVersion (pruning oldest version at the 5-version limit) when it differs from the current definition, instead of skipping. - Bedrock: add a 'bedrock' role_type (bedrock.amazonaws.com trust + S3/KMS least-privilege) and auto-resolve it in BedrockModelBuilder.deploy() for both Nova and OSS paths, replacing the 'role_arn is required' ValueError. - Model package: grant sagemaker:AccessModelPackage (scoped to model-package/*) on the bedrock and serving roles for consuming restricted model packages. - Least-privilege: scope the pipeline sagemaker_policy from Resource '*' to the specific SageMaker resource-type ARNs, and add the required sagemaker:AddTags dependent action. Tests: resolver (policy sync, bedrock role type) and bedrock builder (auto-resolve on Nova/OSS) coverage added/updated. * feat(iam): address second round of PR review on role auto-creation Addresses follow-up review comments on PR #2041: - S3/KMS scoping now accept a single value or a list (Union[str, List[str]]), defaulting to '*'. Each bucket expands to bucket + object ARNs; each KMS key to a key ARN. A list containing '*' collapses to the wildcard. - KMS ARNs are scoped to the caller's account (arn:<p>:kms:*:<account>:key/<id>); region is left '*' to allow cross-region key references. - iam:PassRole resource scoped from '*' to arn:<p>:iam::<account>:role/* (via an IAM_PASSROLE_PLACEHOLDER), keeping the PassedToService=sagemaker condition. - Session fallback uses sagemaker core Session() instead of boto3 directly, so it inherits SDK config/region defaults; dropped the now-unused boto3 import. Tests: list-based S3/KMS expansion, wildcard collapse, account-scoped KMS and PassRole, and the boto-session fallback branches. * feat(iam): log auto-created role and attached/updated policies at WARNING Addresses PR #2041 review: surface account-mutating actions so the user is not taken by surprise. The new IAM role and any policies created, updated, or attached on their behalf are now logged at WARNING (visible by default), while no-op cases (role reuse, already-current policies) stay at INFO. Adds caplog tests for the create/update/no-op paths and the role-creation warning. * feat(train): add Serverless/SMTJ/HyperPod support to trainers and evaluators (#2045) * Add branch for master-nova-reconciliation to workflow * feat(train): add multi-compute backend support (Serverless/SMTJ/HyperPod) to trainers and evaluators * fix gitignore * Addreee PR comments * Restore Compute class name * Update benchmark evaluator * Remove duplicate code * Move training_image out of Compute class * Remove invoke_batch from ModelBuilder * Trainer class fixes * Fixed Trainers and Evaluators; Added CPT Trainer * Clean up comment --------- Co-authored-by: Gokul Anantha Narayanan <166456257+nargokul@users.noreply.github.com> * RLVR reward lambda validation (#2036) * initial commit: copy verifier from nova sdk * feat: rename RFT to RLVR + improve lambda arn checking * feat: update reward verifier + add tests * fix: check lambda arn has "sagemaker" only for nova models * feat: add unit tests + remove sample dataset validation (to be handled by DataPrep APIs) * doc: add example notebook * cleanup: remove misleading unit test * fix: ensure validator is called in rlvr_trainer * feat: Use compute classes in rlvr lambda verification * fix: update rlvr tests to use compute param --------- Co-authored-by: Syed Jafri <syedjfr@amazon.com> * feat: Adding infra validation (#2049) * Add branch for master-nova-reconciliation to workflow * feat: Adding infra validation * fix: make sagemaker session optional param --------- Co-authored-by: Gokul Anantha Narayanan <166456257+nargokul@users.noreply.github.com> Co-authored-by: Syed Jafri <syedjfr@amazon.com> * fix: add hyperpod validation in train and evaluate (#2051) * feat(train): auto-resolve HyperPod recipe from Hub (#2050) * feat(train): auto-resolve HyperPod recipe from Hub Remove the recipe field from HyperPodCompute class since the Trainer and Evaluator already accept recipe as a direct parameter. When recipe is not provided, the SDK now auto-resolves it from SageMaker Hub metadata * update recipe fetching fallback * fix: harden auto-created IAM role security - Add aws:SourceAccount condition to all five auto-role trust policies, scoped to the caller's account at create time, to prevent the cross-service confused-deputy problem. - Narrow iam:PassRole from role/* to role/SageMaker-AutoRole-*, so the auto pipeline role can only pass the SDK's own least-privilege roles. - Tag auto-created roles (CreatedBy, sagemaker:auto-created-role, RoleType) on creation, and backfill tags on reused/pre-existing roles. Add iam:TagRole to the required actions. - Add unit tests covering account scoping, PassRole narrowing, and tagging. * Revert "fix: harden auto-created IAM role security" This reverts commit bb0af9127e7676eae956cbd64c77d5601434d9ac. X-AI-Prompt: Revert the directly-pushed IAM hardening commit from the launch branch so it can be reintroduced via PR X-AI-Tool: kiro-cli * fix: recipe override handling (#2056) * fix: recipe override handling * fix: SMTJ serverful bugs in recipe validation * feat: add Nova as target for LLMAJ (#2059) * wip: add nova specific recipe validations (#2052) * wip: add nova specific recipe validations * feat: Ensure nova specific recipe validations in BaseTrainer * fix: skip nova replicas validation for serverless smtj * cleanup: skip instance type validation for smtj serverless * fix: combine both nova and non-nova validations * fix: integ test bugs * fix: address review comments - renaming variables, updating params etc. * fix: removed replicas check --------- Co-authored-by: Syed Jafri <syedjfr@amazon.com> * feat(train): enable Data Mixing for Nova models#2047 (#2054) * fix: use compute param in get fine tuning utils (#2060) * Hyperpod iam creation (#2057) * feat(iam): add HyperPod IAM support — job-role type and caller-side CLI permission checks Extends the IAM role auto-creation from #2041 to cover the HyperPod training/evaluation path added in #2045. HyperPod has two distinct identities, which this change keeps separate: - Job execution role (assumed by the job ON the cluster, trusted by sagemaker.amazonaws.com): a new "hyperpod" role type carries only job-runtime permissions (S3/ECR/CloudWatch/KMS). It omits the VPC-mode EC2 ENI block, which does not apply to HyperPod cluster jobs. Provision it via TrainDefaults.get_role(role_type="hyperpod"). - Caller identity (runs the HyperPod CLI connect-cluster + start-job locally under its own credentials): needs sagemaker:DescribeCluster, eks:DescribeCluster, eks:AccessKubernetesApi. The SDK cannot auto-create a role for this, so verify_hyperpod_connect_permissions() simulates these against the caller and logs a non-blocking WARNING when any are missing. Wired into both HyperPod submit paths (BaseTrainer._train_hyperpod and BaseEvaluator._submit_hyperpod_eval_job). These are deliberately NOT attached to the SageMaker-trusted job role, which the local caller cannot assume. Also: - _resolve_explicit_role() now accepts role ARNs from any partition (aws-cn/aws-us-gov), not just commercial "aws". - _raise_auto_creation_error() derives the required trust principal from the role config instead of hardcoding sagemaker.amazonaws.com. Tests: - Unit tests for the hyperpod job role (incl. asserting it excludes the CLI connect permissions), the connect-permission verifier, partition handling, and role_type forwarding. - A no-mocks end-to-end integ test that creates the role against real IAM, asserts the job role excludes the connect permissions, checks idempotency, and self-cleans. Verified end-to-end against a live account. * test(iam): add HyperPod connect-permission unit tests + example notebook - Add tests/unit/train/test_hyperpod_connect_permissions.py covering BaseTrainer._train_hyperpod: it verifies the caller's HyperPod CLI connect permissions (forwarding cluster_name), the missing-cluster_name guard fires before that check, and a False/None verdict warns without blocking submit. - Fix a latent bug surfaced by the tests: validate_hyperpod_compute was called in _train_hyperpod but never imported (would NameError on any real run). Added the local import, matching how the evaluator path imports it. - Add v3-examples/hyperpod_iam_role.ipynb: a no-mocks example showing the two HyperPod identities — auto-creating the SageMaker-AutoRole-HyperPod job role, confirming it excludes the CLI connect permissions, and running the caller-side verify_hyperpod_connect_permissions check. Self-cleaning. * refactor(iam): unify compute permission checks behind an infra_type dispatcher Addresses review feedback on the HyperPod IAM changes: 1. Reuse: extract _simulate_denied_actions() in the resolver so both _role_has_sufficient_permissions and verify_hyperpod_connect_permissions share one paginated iam:SimulatePrincipalPolicy implementation instead of duplicating the loop. 2. Imports: hoist the previously-local imports (verify_hyperpod_connect_permissions, validate_hyperpod_compute, _is_nova_model, TrainDefaults) to module top in base_trainer and base_evaluator. No circular-import constraint forced them local. 3+4. Single dispatcher keyed on infra type: add TrainDefaults.get_infra_type(compute) ({serverless, training_job, hyperpod}) and TrainDefaults.resolve_compute_permissions (infra_type, ...). HyperPod verifies caller-side CLI connect permissions and resolves no role; serverless/training_job resolve the training execution role. _train_hyperpod and _submit_hyperpod_eval_job now call this one entry point instead of inlining the split logic. (Per guidance, the shipped resolve_or_create_role(role_type=...) API is left unchanged.) Reverts the temporary role_type param previously added to TrainDefaults.get_role. Tests updated: resolver tests cover _simulate_denied_actions; defaults tests cover get_infra_type + resolve_compute_permissions dispatch; the hyperpod connect tests patch the new dispatcher seam. * fix(notebook): use resolve_or_create_role for HyperPod job role The refactor removed the role_type param from TrainDefaults.get_role, so the notebook's get_role(role_type="hyperpod") call would TypeError. Call resolve_or_create_role(role_type="hyperpod") directly (as the integ test does) and correct the accompanying prose. * refactor(iam): address review — drop dispatcher, partition-aware ARNs, e2e test - Remove get_infra_type and resolve_compute_permissions (the dispatcher was dead code: get_infra_type was never called and the polymorphic ARN-or-None return was a foot-gun). Replace with the intent-revealing TrainDefaults.verify_hyperpod_caller_permissions, which the HyperPod trainer and evaluator paths call directly; serverless/SMTJ continue to use get_role. - Make static policy ARNs partition-aware: _replace_placeholders now rewrites literal arn:aws: resources (e.g. the hyperpod cloudwatch_logs_policy) to the caller's partition, so GovCloud/China/ISO get a matching ARN. No-op for "aws". - Add an end-to-end test (TestTrainHyperPodConnectPermissionsEndToEnd) that runs base_trainer -> real verify helper -> real resolver internals with only the boto IAM/STS clients mocked: a denied connect action warns without blocking submit, and an all-allowed caller produces no warning. Tests updated: resolver tests cover the partition rewrite; defaults tests cover verify_hyperpod_caller_permissions; connect tests patch the new seam. * fix: harden auto-created IAM roles and protect curated recipe keys (#2058) Strengthen the IAM roles the SDK auto-creates in the customer's account and add tamper protection to the training recipe resolver: - Add an aws:SourceAccount condition to all auto-role trust policies, scoped to the caller's account, to prevent the cross-service confused-deputy problem. - Narrow iam:PassRole from role/* to role/SageMaker-AutoRole-*, so the auto pipeline role can only pass the SDK's own least-privilege roles. - Tag auto-created roles (CreatedBy) on creation and backfill on reused roles for attribution and safe cleanup; add iam:TagRole. - Warn when an auto-created role is granted account-wide ("*") S3, KMS, or Glue catalog access, and point to the s3_resource/kms_resource parameters (or supplying an explicit role) to scope it down. - Protect dataset_catalog (alongside model_type and model_name_or_path) from being overridden by user recipes/overrides. - Add unit tests covering all of the above. X-AI-Prompt: Resolve rebase conflicts and squash the IAM role hardening and recipe-key protection changes onto the latest target branch X-AI-Tool: kiro-cli * Recipe validation - unit and integ tests (#2062) * testing: add integ tests for recipe validation * feat: added unit tests for recipe validation * fix: add replicas validation with HyperpodCompute --------- Co-authored-by: Syed Jafri <syedjfr@amazon.com> * fix: set Converse as S3DataType for Nova models in SMTJ Serverful for SFT and DPO (#2064) * fix: hub-content IAM perms, recipe dataset paths, and log markup escaping (#2065) * Add branch for master-nova-reconciliation to workflow * fix: add hub-content IAM perms, inject recipe dataset paths, disable log markup Three independent fine-tuning fixes, each with unit coverage. 1. IAM — DescribeHubContent for fine-tuning roles (sagemaker-core/src/sagemaker/core/helper/iam_policies.py) Fine-tuning jobs reference a base model via a hub-content ARN (e.g. SageMakerPublicHub/Model/<name>/<version>). The SageMaker service reads that hub content as the execution role when it creates the training job, so the role needs sagemaker:DescribeHubContent. Without it, CreateTrainingJob fails with "Access denied to hub content". Added a scoped hub_content_policy (arn:aws:sagemaker:*:*:hub-content/*) to both the "training" and "hyperpod" roles. The account segment is left wildcard so the policy covers the public hub (the "aws" account) and private hubs (SAGEMAKER_HUB_NAME). 2. Recipe — inject resolved dataset channel paths (sagemaker-train/src/sagemaker/train/base_trainer.py) _train_serverful_smtj rendered the recipe template with empty data.train_files / data.val_files, so the training container aborted. Now the resolved training/validation datasets are mapped to their container channel mount paths (/opt/ml/input/data/<channel>) and injected into the recipe override spec before rendering. An S3 prefix maps to the channel directory; a trailing object key with an extension maps to the mounted file. Datasets are resolved once up front (the later resolution was removed) and the trainer environment is now forwarded to ModelTrainer.from_recipe. 3. Logs — disable rich markup for streamed container log lines (sagemaker-core/src/sagemaker/core/tools/templates.py + regenerated sagemaker-core/src/sagemaker/core/resources.py) Container log lines are arbitrary text and may contain square brackets (e.g. file paths like [.../main_ppo.py]). The rich logging handler parsed them as markup tags and raised MarkupError, aborting the wait loop. The PRINT_WAIT_LOGS codegen template now passes extra={"markup": False} when logging streamed events; resources.py was regenerated via codegen so the change lands in TrainingJob, ProcessingJob, and TransformJob wait() loops. The generated file was NOT hand-edited — the template is the source of truth. Tests - test_iam_role_resolver.py: training and hyperpod roles carry DescribeHubContent scoped to hub-content/*. - test_base_trainer_serverful.py (new): S3 prefix vs object-key channel mount mapping, validation-absent case, in-place spec mutation, and environment forwarding. - test_resources.py (TestWaitLogMarkup): TrainingJob/ProcessingJob/ TransformJob wait() log streamed lines with extra={"markup": False}. * fix: Evaluation on hyperpod (#2061) * fix: Evaluation on hyperpod Fixed E2E Benchmark and custom evaluation experience using trained models on hyperpod * fix endpoint deployment * bug fixes in evaluator validation * address PR comments * fix: skip model_package_group validation when HyperPod compute is provided in CPTTrainer (#2068) _validate_and_resolve_model_package_group() does not accept a 'compute' kwarg. Guard the call with 'if compute is None' to match the pattern used by SFT, RLVR, and DPO trainers. * fix: resolve HyperPod training image from EKS payload template (#2069) For HyperPod-only techniques like CPT (non-datamixing), get_training_image() looks at SmtjImageUri which is not populated for HyperPod recipes. This caused jobs to be submitted without a container image, leading to PyTorchJob validation failures. Updated _train_hyperpod image resolution: 1. User-provided training_image 2. get_training_image() (SmtjImageUri) with SM-TJ -> SM-HP tag swap 3. Fallback: get_hyperpod_training_image() extracts image from template 4. Raise ValueError if still unresolved Added extract_image_from_hyperpod_template() as shared helper, refactored data_mixing_utils to use it. Added unit tests. * fix: apply recipe overrides to hyperparameters in SMTJ serverful path (#2070) The _train_serverful_smtj code path was not calling _apply_recipe_to_hyperparameters(), so user-provided overrides (e.g. overrides={'max_epochs': 1, 'name': '...'}) were silently dropped from the hyperparameters dict passed to ModelTrainer.from_recipe(). Added the same _apply_recipe_to_hyperparameters() call that the serverless path already uses, placed after subclass extra HP injection. Unit tests added to verify overrides overwrite base HP values and coexist with subclass-injected extra hyperparameters. * feat: support serverful training job checkpoint resolution in InspectAI evaluator (#2066) * fix: reject unknown recipe overrides (serverless + serverful) and untrusted IAM roles (#2071) * fix: drop unknown recipe overrides and reject untrusted IAM roles Two bug-bash fixes for the Nova reconciliation flow. 1. Recipe overrides (SFT and other trainers) Override keys that don't exist in the base recipe were silently merged into the resolved recipe. Now RecipeResolver drops any override/user-recipe key with no counterpart in the base recipe (recursing into nested mappings, preserving known siblings) and logs a warning naming the dropped key and its source. Power-user overrides of non-spec keys that DO exist in the recipe still apply. 2. IAM role resolution (auto-create fallback) resolve_or_create_role reused the caller's role whenever it passed the permission simulation, without checking the role's trust policy. An admin role with all permissions but no sagemaker.amazonaws.com trust was handed to CreateTrainingJob, which failed with "Could not assume role". The resolver now also verifies the role trusts the required service principal; on a definitive "not trusted" it warns and falls back to the auto-created role. Adds unit tests covering both behaviors. * fix: drop unknown recipe overrides on serverful SMTJ path Bug 3: on the serverful SMTJ recipe path (ModelTrainer.from_recipe -> _load_base_recipe), override keys that don't exist in the base recipe (e.g. max_steps for a model whose recipe has no such field) were merged in unvalidated with no warning. _load_base_recipe now filters recipe_overrides against the loaded base recipe before the OmegaConf merge: any key (top-level or nested) with no counterpart in the recipe is dropped and a warning naming it is logged, while valid overrides and known siblings are preserved. This mirrors the drop-and-warn behavior already applied in RecipeResolver for the serverless path (bug 18). Adds unit tests for _load_base_recipe and the _drop_unknown_recipe_overrides helper. * test: use placeholder account and generic role name in IAM tests Address PR feedback: replace the real AWS account 618100645563 with the documentation placeholder 123456789012, and the Amazon-specific role name IibsAdminAccess-DO-NOT-DELETE with a generic 'Admin' role in the IAM role resolver unit tests. * Fix recipe override errors in evaluator (#2067) * fix: Evaluation on hyperpod Fixed E2E Benchmark and custom evaluation experience using trained models on hyperpod * fix endpoint deployment * bug fixes in evaluator validation * address PR comments * fix: custom scorer evaluators Fix recipe override handling * extend evaluators to support OSS models * Fix custom eval for OSS model * Fix custom eval and add unit tests * fix unit tests failures * fix: MLFlow error causing OSS model eval to fail (#2075) * fix: Evaluation on hyperpod Fixed E2E Benchmark and custom evaluation experience using trained models on hyperpod * fix endpoint deployment * bug fixes in evaluator validation * address PR comments * fix: custom scorer evaluators Fix recipe override handling * extend evaluators to support OSS models * Fix custom eval for OSS model * Fix custom eval and add unit tests * fix unit tests failures * fix: mlflow key name error causing non-Nova eval to fail * update mlflow fix * Add mlflow flields to SMTJ serverful * fix(train): use Converse S3DataType for Nova SFT/DPO in serverless flow (#2079) The serverless training path hardcoded s3_data_type="S3Prefix" for all models in _convert_input_data_to_channels. For Nova SFT/DPO multimodal datasets this must be "Converse" so the SageMaker data agent downloads images and rewrites "uri" to "localPath" in the JSONL. Without this, the container validator fails with "Invalid S3Location, 'uri' is required". Add s3_data_type parameter to _convert_input_data_to_channels (default "S3Prefix" for backward compat) and pass "Converse" from sft_trainer and dpo_trainer when the model is Nova — matching the serverful SMTJ fix. * [Bugfix] RLVR Setup and Reward Lambda handling (#2076) Co-authored-by: Ealynn Hsu <ealynnh@amazon.com> * fix(train): route .hyperparameters.* through recipe resolver (#2078) * fix(train): route .hyperparameters.* through recipe resolver on serverless Remove early-return guard in _apply_recipe_to_hyperparameters so user-set values reach get_resolved_recipe_from_context, which promotes them into synthetic overrides the service will honor. * Update Base Evaluator * fix resolved recipes * Fix: Check s3 permission before saving recipe.yaml (#2083) * Fix: Check s3 permission before saving recipe.yaml * Adding unit tests * Address comments --------- Co-authored-by: Ealynn Hsu <ealynnh@amazon.com> * Bugfix: get_resolve_recipe() includes all overrides and displays as a nested dictionary (#2084) * WIP: get_resolved_recipe structure fixes * WIP: recipe resolution * patch output and data s3 paths in recipe preview * Unit tests and adding the logger properly --------- Co-authored-by: Ealynn Hsu <ealynnh@amazon.com> * fix(train): validate recipe and instance count override for SMTJ serverful (#2082) Co-authored-by: Mahima Chaudhary <mahchy@amazon.com> * fix(iam): validate roles by default, opt-in creation, and add MLflow perms (#2080) * fix(iam): validate roles by default, opt-in creation, and add MLflow perms Security hardening of IAM role handling plus MLflow support for training/eval. Validate-by-default (no silent auto-creation): - resolve_or_create_role is replaced by read-only resolve_and_validate_role, which resolves the provided/caller role and validates permissions + trust via iam:SimulatePrincipalPolicy. It never creates or mutates IAM. A missing permission raises RoleValidationError listing the exact missing actions and remediation; an unverifiable verdict (caller lacks SimulatePrincipalPolicy) warns and proceeds so Studio/notebook flows are not broken. - Explicit, opt-in creation moves to the IamRoleResolver class (iam_role_creator.py): create_execution_role / delete_execution_role / get_required_actions. This is the only code path that writes IAM. - All call sites (training defaults, evaluator, ModelBuilder, Bedrock, Pipeline, feature_store scheduler) switched to validate-only. MLflow permissions: - Add mlflow_policy to the training role (covers eval, which resolves role_type="training"). Enumerated sagemaker-mlflow data-plane actions only (no AccessUI, no destructive/control-plane actions), scoped to mlflow-app/* and mlflow-tracking-server/*. Training/eval containers log to managed MLflow as the execution role via the sagemaker-mlflow SigV4 plugin. Review fixes: - Validation gates only on *-resource "smoke test" actions; resource-scoped actions (mlflow/hub/model-package) are excluded so simulating without ResourceArns can't produce false denials (and non-MLflow jobs aren't blocked). - Add the aws:SourceAccount confused-deputy condition to the hyperpod trust policy (was the only role missing it). Tests + docs: reworked unit tests (validate-only + creator), new integ test for the validation path, and an iam_role_validation.ipynb example; deprecation banners on the old auto-creation notebooks. * fix(iam): address PR feedback — ListHubs, full MLflow set, validate provided roles - Add sagemaker:ListHubs to the hub_content_policy (training + hyperpod) so the hub-content resolution path can list hubs. - Expand mlflow_policy to mirror the Nova Forge SDK's documented MLflowSageMaker policy (nova-customization-sdk/docs/iam_setup.md): adds the experiment-tracking, logged-model, and model-registry actions (Create/Update RegisteredModel + ModelVersion, FinalizeLoggedModel, LogInputs/Outputs, Get*/Search*, AccessUI, Restore*, tag actions). Nothing removed; still scoped to mlflow-app/* and mlflow-tracking-server/*. - Validate user-PROVIDED roles, not just the caller fallback: ModelBuilder (build/deploy paths), Bedrock, and Pipeline now pass the user's role into resolve_and_validate_role instead of only calling it when no role was given. Construction-time ModelBuilder still only resolves a default when none is supplied (no IAM work when a role is provided). Train + feature_scheduler already threaded the provided role. - Drop the "never auto-created" phrasing from comments/docstrings/error text. - Update affected unit tests (serve, mlops) to stub resolve_and_validate_role on the now-validated provided-role paths. Fix a stray typo in the example notebook. All affected unit suites pass: core 76, train 233, serve 126, mlops 69. * fix(iam): address PR review — merge creator into resolver, MLflow describe, cleanups - Merge iam_role_creator.py into iam_role_resolver.py: the IamRoleResolver class (create/delete/get_required_actions) now lives alongside the read-only resolve_and_validate_role in the resolver module, so the class name matches the file. Deleted iam_role_creator.py; helper __init__ exports IamRoleResolver from the resolver module. (review: class name should match file) - mlflow_policy: add a control-plane describe statement (sagemaker:DescribeMlflowApp + DescribeMlflowTrackingServer) scoped to mlflow-app/* and mlflow-tracking-server/*, so a job can resolve the tracking endpoint from a provided MLflow ARN. (review: is DescribeMlflowTrackingServer needed when a user provides an MLflow app — yes) - create_execution_role: call get_caller_identity once. (review nit) - Delete the deprecated example notebooks iam_role_autocreation.ipynb and hyperpod_iam_role.ipynb (auto-creation removed; iam_role_validation.ipynb replaces them). (review: are these still in use) IAM unit suite: 76 passed. * fix(iam): add MLflow permissions to the hyperpod execution role MLflow applies to HyperPod jobs: when an mlflow_resource_arn is provided, the tracking URI/experiment/run are injected into the HyperPod recipe (base_trainer._train_hyperpod), and the job logs to managed MLflow on the cluster as its execution role — same as the SMTJ/serverless training path. The hyperpod role config was missing mlflow_policy, so add it (data-plane sagemaker-mlflow logging + control-plane DescribeMlflowApp/DescribeMlflowTrackingServer), mirroring the training role and scoped to mlflow-app/* and mlflow-tracking-server/*. Tests updated: assert MLflow perms on both training and hyperpod (the execution-role types that log to MLflow) and absent elsewhere. 77 passed. * fix(iam): add lambda_policy to hyperpod role (RLVR reward Lambda parity) Audit of training vs hyperpod execution-role policies found one more gap beyond MLflow: RLVR supports a Lambda custom_reward_function and runs on HyperPod (HyperPodCompute -> _train_hyperpod). When the reward function is a Lambda ARN, the job invokes it as the execution role, so the hyperpod role needs lambda:InvokeFunction — which only the training role had. Added it (mirrors training; function:* resource). Other training/hyperpod diffs were reviewed and are intentional: model_package (HyperPod doesn't register model packages), ec2 ENI (VPC-mode SMTJ only), and the cloudwatch_logs resource prefix (HyperPod logs aren't under /TrainingJobs*). 78 unit tests passed. * fix(iam): grant evaluation lineage + training-job permissions to the training role The evaluation pipeline runs as the training execution role (SageMaker-AutoRole- Training) and failed two steps: - CreateEvaluationAction: missing sagemaker:CreateAction on action/* - EvaluateCustomModel: missing sagemaker:AddTags on training-job/* (the existing model_package AddTags was scoped to model-package* resources only) Add an evaluation_policy to the training role covering the lineage operations the eval flow performs (Action/Artifact/Context create+describe+associations, scoped to action/*, artifact/*, context/*) and the training-job launch+tag (CreateTrainingJob/DescribeTrainingJob/StopTrainingJob/AddTags on training-job/*). 79 unit tests passed. * feat(train): add iterative training with base_model_name param (#2085) Add support for iterative training (resuming from S3 checkpoints) Co-authored-by: Mahima Chaudhary <mahchy@amazon.com> * Address PR #2074 review comments: codegen-emitted model_artifacts synthesis + tests (#2086) * feat: add support for Bedrock deployments from S3 paths * Address PR #2074 review comments - Emit TrainingJob.get() model_artifacts synthesis from the codegen engine (resources_codegen._get_get_method_post_processing + GET_METHOD_TEMPLATE {post_processing} hook) so the customization survives the next autogeneration of resources.py. - Add unit tests for the TrainingJob.get() model_artifacts synthesis (TestTrainingJobGetModelArtifactsSynthesis). - Add regression tests for trainer-checkpoint resolution from _latest_training_job.model_artifacts in BaseEvaluator._submit_hyperpod_eval_job and InspectAIEvaluator._resolve_trainer_model. * Fix hardcoded service-2.json path in data_extractor; regenerate resources.py - Remove the hardcoded SERVICE_JSON_FILE_PATH override pointing at a foreign developer's machine; fall back to the package-relative path already imported from constants (_PACKAGE_ROOT-based), so codegen runs from any clone. - Regenerate resources.py: the TrainingJob.get() model_artifacts synthesis block is emitted identically by the codegen engine; black (pinned 24.3.0) collapses the ModelArtifacts(...) call to one line. Confirmed idempotent and that config_schema.py / shapes.py regenerate with no changes. * Rename _get_get_method_post_processing to _get_method_post_processing Address review comment: drop the doubled 'get_get' in the codegen helper name. Updates the definition, call site, and the test-file docstring reference. --------- Co-authored-by: Lisa Ni <nilis@amazon.com> * release: update version and changelogs * fix: update changelog * fix: simplify subscription recipe handling and fix unit tests (#5973) Remove duplicate {customer_id} replacement and redundant subscription recipe override_params merge logic. Update tests to mock resolve_and_validate_role and get_hyperpod_recipe_path introduced in recent IAM/recipe validation changes. Co-authored-by: Vaishnavi Kommaraju <vvak@amazon.com> * fix: remove unused training_image tests (#5972) * fix: add training_image param to Compute class * fix: remove non-existing training image field from Compute unit tests --------- Co-authored-by: Syed Jafri <syedjfr@amazon.com> * Update documentation and unit tests (#5971) * fix: recipe unit tests to use jumptstart tags (#5974) * fix: add training_image param to Compute class * fix: remove non-existing training image field from Compute unit tests * fix: use get_jumpstart_tags in unit tests --------- Co-authored-by: Syed Jafri <syedjfr@amazon.com> --------- Co-authored-by: Gokul Anantha Narayanan <166456257+nargokul@users.noreply.github.com> Co-authored-by: Syed Jafri <syedjfr@amazon.com> Co-authored-by: LN <133025223+amazeAmazing@users.noreply.github.com> Co-authored-by: Luke Luneau <luluneau@amazon.com> Co-authored-by: Zhaoqi <jzhaoqwa@amazon.com> Co-authored-by: Lucas Jia <lucasjia@amazon.com> Co-authored-by: Ealynn Hsu <89547630+ehsu3@users.noreply.github.com> Co-authored-by: Ealynn Hsu <ealynnh@amazon.com> Co-authored-by: Mahima Chaudhary <36598677+cmahima@users.noreply.github.com> Co-authored-by: Mahima Chaudhary <mahchy@amazon.com> Co-authored-by: Lisa Ni <nilis@amazon.com> Co-authored-by: vaishnavi-kommaraju <vaishnavi.k14@gmail.com> Co-authored-by: Vaishnavi Kommaraju <vvak@amazon.com>Syed Mujtaba · dba1127a · 2026-06-27
- 9.3ETVMaster nova follow ups (#6051) * Feat: show_metrics() and stream_logs() helper functions (#6002) * Feat: Add show_metrics() and stream_logs() for monitoring training jobs * Feat: show_metrics() and stream_logs() --------- Co-authored-by: Ealynn Hsu <ealynnh@amazon.com> * show_metrics() Enhancement: Display MLFlow metrics for OSS models (#6013) * show_metrics() Enhancement: Display MLFlow metrics for OSS models * Update unit tests * Address code comments --------- Co-authored-by: Ealynn Hsu <ealynnh@amazon.com> * feat(serve): add opt-in model source tag-based resource reuse (#5993) * feat(serve): add opt-in model source tag-based resource reuse Add reuse_resources to ModelBuilder.build/deploy and BedrockModelBuilder.deploy. On a hit, discover an existing resource by the model-source tag and return it instead of creating a duplicate (warn, do not raise). Honored per call. - New sagemaker/serve/model_reuse.py: tag helpers + service-client discovery - Consolidate Nova manifest/checkpoint reading into sagemaker/core/training/utils.py - SageMaker: build() skips Model creation on reuse (sets built_model to the existing Model); deploy() reuses the endpoint after validating env vars/image/ instance type (PrimaryContainer with Containers[0] fallback for Nova) - Reuse gates are skipped for inference-component builds/deploys so IC create/update (via _deploy_for_ic) is never silently intercepted - Bedrock: reuse custom model + active deployment; response includes modelArn - Reuse discovery uses the cached session/bedrock clients - Support raw S3 URI model input via model_metadata BASE_MODEL_NAME - Unit tests + notebook examples * feat(serve): support Nova inference-component deployment and harden reuse Route Nova model-customization deploys through the shared single-inference -component path when a ResourceRequirements inference_config is supplied, so each Nova checkpoint (full-rank or LoRA-merged) is hosted as one inference component referencing the built Model. Nova without an inference_config keeps the direct model-on-variant path. - Broaden _is_nova_model to identify Nova from a package-less source (raw S3 checkpoint or trainer) via base_model_name, in addition to the model package recipe/hub-content name. - Set EnableNetworkIsolation on the IC endpoint config to match the built Model (always True for Nova), fixing CreateInferenceComponent rejection on mismatched network isolation. - Guard model-package-dependent logic (restricted-package path, PEFT/recipe metadata, lineage tracking) so package-less Nova checkpoints deploy cleanly. - Apply accumulated tags (including the model-source reuse tag) to endpoints created on the shared IC path so they remain discoverable. - build(reuse_resources=True) only short-circuits when the backing Model can be resolved; IC endpoints and stale/deleted configs fall through and build a real Model, preventing a None built_model on later IC deploys. - deploy() warns that reuse_resources has no effect for inference-component deployments, which manage their own reuse by component name. - Surface both the manifest.json and output.tar.gz errors when Nova checkpoint URI resolution fails, instead of masking the primary failure. Add unit tests covering the Nova IC path (routing, network isolation, IC spec) and the model-on-variant fallback. * fix(core): resolve Nova checkpoint manifest across all three output layouts Nova training jobs write their checkpoint manifest to different locations depending on the training platform: HyperPod: <output>/<job>/manifest.json Serverless: <output>/<job>/output/output/manifest.json Serverful: <output>/<job>/output/output.tar.gz (manifest inside) resolve_nova_checkpoint_uri previously only tried the serverless manifest path and the serverful tar.gz, so HyperPod jobs (manifest directly under the job directory) failed to resolve. Add build_nova_hyperpod_manifest_s3_uri and try all three layouts in turn, aggregating every failure into the raised error so the real cause is not masked by the last attempt's message. Add unit tests for the HyperPod builder and for resolution from the HyperPod and serverless layouts. * feat(serve): Model-tag reuse, IC-deploy guard, and instance_type fix Simplify reuse discovery by tagging SageMaker Models (not just endpoints) with the model-source identifier, so build(reuse_resources=True) can find and skip recreating an existing Model directly — no IC-state dependency. - Tag non-Nova Models at build time with the model-source tag (matching the Nova path's existing behavior). Both Nova and OSS Models are now discoverable by tag. - Add _find_reusable_model: build(reuse_resources=True) searches Models by source tag, skipping Model creation on a hit. Also discovers the endpoint for deploy() to reuse later. - Simplify _get_model_for_endpoint back to variant-only lookup (returns None for IC endpoints). No longer needs IC-spec resolution since the Model is found directly by tag. - _reused_endpoint_matches_config returns True for IC endpoints (can't read container config from variant; Model was already matched by tag). - deploy() with reuse_resources=True on an IC deploy logs a warning that the flag has no effect (ICs manage reuse by endpoint_name + IC name). - Fix deploy() to set self.instance_type from the caller's explicit value before calling _deploy_model_customization, preventing recipe-resolved defaults from overriding the user's intent. - Add model-source tag assertion to the existing OSS deploy integ test. * feat(train): add dry_run=True to train() (#6027) * feat(train): add dry_run=True to train() Add dry_run parameter to all trainers (SFT, DPO, RLVR, RLAIF). When dry_run=True: - All existing validation runs inline (IAM role, hyperparameters, recipe constraints, infrastructure availability) - Returns None without submitting a job or consuming compute - Raises with clear error message on validation failure Additionally, validate_data_path_exists() is called unconditionally (regardless of dry_run) before job submission to catch non-existent S3 paths or dataset ARNs early. Design follows nova-forge-sdk pattern: validation always runs as part of the normal code path, dry_run short-circuits before the actual TrainingJob.create API call. Changes: - data_utils.py: add validate_data_path_exists() utility (S3 + DataSet ARN) - base_trainer.py: add dry_run to abstract train(), _train_serverful_smtj(), and _train_hyperpod() - sft/dpo/rlvr/rlaif_trainer.py: add dry_run param, pass through to shared methods, short-circuit serverless path - Notebook examples added to SFT, DPO, RLVR, RLAIF notebooks - Unit tests added to existing test files - Integration test added * feat(evaluate): add dry_run=True to evaluate() Add dry_run parameter to BaseEvaluator.evaluate() and all subclasses (BenchMarkEvaluator, CustomScorerEvaluator, LLMAsJudgeEvaluator). When dry_run=True: - All existing validation runs (IAM role, model resolution, recipe, pipeline rendering) - Dataset S3 path / DataSet ARN validated via validate_data_path_exists() - Returns None without submitting a pipeline execution - Raises on validation failure Dataset validation runs unconditionally (not just during dry_run) for CustomScorerEvaluator and LLMAsJudgeEvaluator which accept user datasets. Changes: - base_evaluator.py: add dry_run to evaluate() signature - benchmark_evaluator.py: add dry_run, short-circuit before _start_execution() - custom_scorer_evaluator.py: add dry_run, validate dataset, short-circuit - llm_as_judge_evaluator.py: add dry_run, validate dataset, short-circuit - Notebook examples added to benchmark, custom_scorer, llm_as_judge notebooks * fix(dry_run): support DataSet objects, deduplicate ARN validation, expand coverage - validate_data_path_exists() now accepts Union[str, DataSet]; extracts .arn from DataSet objects for validation - Removed duplicate ARN validation logic; delegates to _validate_dataset_arn_exists() - _validate_dataset_arn_exists() warns on AccessDenied instead of raising (execution role may still have access) - Removed isinstance(..., str) guards in all trainers and evaluators so DataSet objects flow through validation - Added dry_run=True parameter to CPTTrainer.train() - Added dry_run=True parameter to ModelTrainer.train() - Integration test: valid_dataset fixture no longer re-creates on every run - Integration test: added nonexistent_dataset_arn and nonexistent_dataset_obj fixtures - Integration test: added TestDryRunServerful class (serverful compute path) - Unit test: added test_dataset_object_extracts_arn, test_dataset_object_not_found_raises * fix(dry_run): support all AWS partitions in DataSet ARN validation - Update ARN regex to aws(?:-[a-z]+)* to match aws-cn, aws-us-gov, aws-iso, aws-iso-b partitions - Use regex guard in validate_data_path_exists() for consistent matching - Add unit tests for each partition (standard, China, GovCloud, ISO, ISO-B) and invalid partition rejection * [Feat]: Job Notifications for SMTJ (#6042) * [WIP] Job notifications setup * [WIP] Job notif update * Adding tests, dedupe logic, and example * Update ARN example values, clean up SM session definitions, rename func * Update trainers to include 'notifications' param and add arn regex check * Address PR comments, use botocore errors, return notification rule arn --------- Co-authored-by: Ealynn Hsu <ealynnh@amazon.com> * [Docs] Add documentation for show_metrics(), stream_logs(), and job notification setup (#6065) * docs: Add show_metrics, stream_logs, and job notifications documentation * Update example job names * Move monitoring capabilities to model_customization --------- Co-authored-by: Ealynn Hsu <ealynnh@amazon.com> * documentation: add dry-run and resource reuse docs to existing RST pages (#6061) * feat(evaluate): add dry_run and caller IAM permission validation to a… (#6075) * feat(evaluate): add dry_run and caller IAM permission validation to all evaluators - Add dry_run=True parameter to InspectAIEvaluator and MultiTurnRLEvaluator (Benchmark, CustomScorer, LLMAsJudge already had it) - Add verify_evaluation_caller_permissions() to validate the caller's identity has the pipeline-orchestration permissions before submitting - Define EVALUATION_CALLER_ACTIONS constant with the set of 22 IAM actions needed by whoever calls evaluator.evaluate() - Fix bug: LLMAsJudge InspectAI code path was missing the dry_run check - Wire dry_run through _get_aws_execution_context() for all evaluators - Add unit tests covering dry_run behavior across all evaluator types * change: move EVALUATION_CALLER_ACTIONS to iam_policies.py * fix: SMHP RLVR image selection and storm_rbs recipe cleanup (#6079) * [Fix] Remove task-type from RLVR recipe, update RLVR image selection logic * Add check for RLVR/RFT before removing task_type * fix: prefer SMHP image over SMTJ fallback in _train_hyperpod In _train_hyperpod, try get_hyperpod_training_image first (native SMHP image) and only fall back to SMTJ image with SM-TJ->SM-HP tag replacement if the SMHP image is not available. Previously the order was inverted. --------- Co-authored-by: Ealynn Hsu <ealynnh@amazon.com> * fix: stream_logs_smhp extract training job from obj (#6084) Co-authored-by: Syed Jafri <syedjfr@amazon.com> * Fix mlflow (oss models) metrics viz (#6102) * fix: stream_logs_smhp extract training job from obj * fix: render mlflow metrics as png to handle large number of metrics * code cleanup: move io, base64 to top level imports --------- Co-authored-by: Syed Jafri <syedjfr@amazon.com> * fix(dry_run): skip MLflow app creation during dry_run (#6100) When dry_run=True, _resolve_mlflow_resource_arn now lists existing apps but skips creation and waiting. All trainers (SFT, DPO, RLVR, RLAIF, MultiTurnRL) forward dry_run to MLflow resolution. 9 unit tests. * Update error message on ModelBuilder when deploying from S3 checkpoint (#6111) * Update error message on ModelBuilder when deploying from S3 checkpoint * Update ModelBuilder to automatically find image_uri * Update import and methods * Add create notifications helper method (#6113) * fix: stream_logs_smhp extract training job from obj * fix: render mlflow metrics as png to handle large number of metrics * code cleanup: move io, base64 to top level imports * fix: add helper method to create sns topic * fix: renamed IDs for readability in SNS access policy --------- Co-authored-by: Syed Jafri <syedjfr@amazon.com> * fix(serve): BedrockModelBuilder accepts BaseTrainer as model input (#6104) BedrockModelBuilder(model=trainer) now works for SFTTrainer, DPOTrainer, RLVRTrainer, etc. Previously only ModelTrainer and S3 URIs were supported. * fix(serve): fix two P0 resource-reuse bugs (instance-type reuse + Bedrock permission fail-open) (#6109) Two independent reuse defects surfaced during Nova SDK dogfooding, both under reuse_resources=True. Fixed together as they share the reuse code paths. --- 1. instance_type mismatch silently reused --- deploy() trusted the endpoint candidate cached by build() in _reused_endpoint_name without re-validating it. That cache is resolved at build time, before instance_type (a deploy() argument) is known, so a deploy on a different instance type reused the wrong endpoint. Separately, _reused_endpoint_matches_config skipped the instance_type check entirely for Inference Component (IC) endpoints: the IC early-return (no ModelName on the variant) returned True before reaching the check. deploy() now re-validates the cached endpoint against the requested instance_type and falls back to a fresh discovery on a miss or mismatch. _reused_endpoint_matches_config checks instance_type -- which lives on the production variant and is available for every endpoint, including IC ones -- before the IC early-return, so an instance-type mismatch is never silently reused. --- 2. Bedrock reuse_resources=True fails open on missing permission --- Reuse discovery (find_existing_bedrock_model, find_active_bedrock_deployment_for_model, find_existing_sagemaker_endpoint) swallowed every exception and returned None. When the execution role lacked a read permission (e.g. bedrock:ListTagsForResource), a denied discovery call was indistinguishable from "nothing to reuse", so reuse fell through to creating the resource -- which then failed with a confusing "ValidationException: Model with name '...' already exists" because the prior run's resource still held the deterministic name, never surfacing the real cause. A new _reraise_if_access_denied helper re-raises AccessDeniedException as a PermissionError naming the missing IAM action; all other errors still fail open (warn and return None) so transient failures like throttling do not block a deploy. Tests: 86 unit tests pass across test_model_builder.py and test_model_reuse.py, including regression guards for both fixes (IC instance-type mismatch, build-time cache re-validation, access-denied-raises, and fail-open-on-other-error). Co-authored-by: Elise Harvey <harveel@amazon.com> * fix(train): raise on expired credentials in show_metrics log fetch (#6114) * fix(train): raise on expired credentials in show_metrics log fetch When AWS credentials expire, show_metrics() reported "No CloudWatch logs found for job '<name>'. The job may still be starting, or logs may not be available yet", sending users to debug their training job instead of refreshing credentials. _fetch_smtj_logs() and _fetch_smhp_logs() each wrapped their CloudWatch Logs call in a bare `except Exception`, logged a warning, and returned an empty list, so an ExpiredTokenException was indistinguishable from a job that genuinely has no logs yet. The empty list then became the misleading ValueError above. Design follows the existing notifications.py pattern: read the structured error code, re-raise the auth subset as PermissionError with remediation guidance, and let every other code keep its current degrade-to-empty behavior. A genuinely absent log group (ResourceNotFoundException) still returns an empty list, so a just-started job continues to raise the existing "No CloudWatch logs found" ValueError. Changes: - cloudwatch_metrics.py: add _AUTH_ERROR_CODES and _raise_if_auth_error(); apply at describe_log_streams, get_log_events, and filter_log_events; narrow `except Exception` to `except ClientError` - base_trainer.py: document PermissionError on show_metrics() - test_cloudwatch_metrics.py: 20 unit tests covering all 7 auth codes on both platforms, ResourceNotFoundException degradation, mid-pagination failure, and a regression guard for the existing "no logs found" path * change(train): move AUTH_ERROR_CODES to common_utils/constants.py Addresses review feedback: the auth error-code list was duplicated in cloudwatch_metrics.py and its unit test, so the two lists had to be kept in sync manually. Move it to the existing common_utils/constants.py as a module-level frozenset and import it in both places. Test parametrization sorts the frozenset so test IDs stay deterministic. --------- Co-authored-by: sayemkam <sayemkam@amazon.com> * fix: serverful instance type validations + integ tests (#6124) * fix: serverful instance type validations + integ tests * cleanup: add missing newlines to end of files * style(sagemaker-train): Fix missing trailing newlines * fix(train): Log SMHP enum fetch failure at debug, not warning + added unit tests --------- Co-authored-by: Syed Jafri <syedjfr@amazon.com> * fix: update job key used for show_metrics/stream_logs in MTRL (#6128) * Update error message on ModelBuilder when deploying from S3 checkpoint * Update ModelBuilder to automatically find image_uri * Update import and methods * Improving logging level * fix: update job key used for show_metrics/stream_logs in MTRL * Stream logs unified improvement (#6127) * fix(stream_logs): unify log streaming for MTRL, evaluators, and HyperPod Fixes three stream_logs() bugs identified in bug bash testing: 1. MTRL trainer stream_logs() now uses the correct log group (/aws/sagemaker/Job/AgentRFT) and polls status via Job API instead of TrainingJob API — previously hung forever showing nothing. 2. Adds stream_logs() to BaseEvaluator and EvaluationPipelineExecution with support for pipeline, MTRL eval, and HyperPod backends. 3. Patches _stream_logs_smhp() to provide user feedback instead of silently swallowing ResourceNotFoundException and empty events. Introduces LogStreamer utility (poll-once pattern) and stream_log_loop() shared helper to eliminate code duplication across all callers. * test(stream_logs): add evaluator integ tests using existing completed jobs Integration tests for evaluator.stream_logs() against completed pipeline executions in us-west-2/729646638167. Covers BenchMarkEvaluator, CustomScorerEvaluator, and LLMAsJudgeEvaluator. No new jobs launched. * Improve logging, error messages and minor bug fixes (#6135) * Update error message on ModelBuilder when deploying from S3 checkpoint * Update ModelBuilder to automatically find image_uri * Update import and methods * Improving logging level * fix: update job key used for show_metrics/stream_logs in MTRL * fix(serve): Speed up reuse_resources with Tagging API and resolve string training jobs Use resourcegroupstaggingapi.get_resources() for O(1) tag lookups instead of scanning all models/endpoints, and auto-resolve string _latest_training_job to TrainingJob objects in ModelBuilder so trainers work without manual .get(). * Remove region logging * fix(train): Improve trainer UX and reduce verbose logging - Cache sagemaker_session in BaseTrainer.__init__ to avoid creating duplicate sessions on every method call - Remove redundant role validation (was validating 3x per train() call), now validates once in ModelTrainer.__init__ - Demote noisy INFO logs to DEBUG (role validated, stopping condition defaults, recipe paths, output compression) - Add num_lines param to stream_logs() to limit output for long jobs - Prefix CloudWatch log lines with [CloudWatch] and use print() to distinguish container output from SDK logging - Improve show_metrics() error message when time range yields no logs - Raise ValueError on AccessDenied in dry_run data path validation instead of silently warning - Improve model_package_group error message to mention compute option - Move local imports in _train_serverful_smtj to top-level - Remove redundant get_role() call in ModelTrainer.from_recipe() * resolve conflict * Update import * fix(serve): Move endpoint reuse discovery to deploy time (#6142) * fix(serve): Move endpoint reuse discovery to deploy time * cleanup: Remove stray dev logs --------- Co-authored-by: Syed Jafri <syedjfr@amazon.com> * fix: usuability update from feedback (#6150) * fix: usuability update from feedback * update documentation * Doc update: reorganize examples folder (#6155) * fix: usuability update from feedback * update documentation * docs: organize model-customization examples into serverless/serverful/deployment/evaluation subfolders * add job notification integ test * feat: bedrock model reuse for OSS models (#6157) * feat: bedrock model reuse for OSS models * code cleanup: fixed log statements, renamed variables, added type checks --------- Co-authored-by: Syed Jafri <syedjfr@amazon.com> * test: add integ tests for show_metrics and model reuse (reuse_resources=True) (#6158) - Add show_metrics() assertions to existing Nova SFT tests (serverless + serverful) - Add show_metrics() via MLflow to OSS Llama SFT test - Add show_metrics() to MTRL trainer integration test - Add reuse round-trip tests to test_model_customization_deployment.py (OSS, us-west-2) - Add reuse round-trip tests to test_nova_model_customization_deployment.py (Nova, us-east-1) - Add build(reuse_resources=True) tests to both deployment test files * feat(telemetry): Add tracking for dry_run, notifications, reuse (#6159) * fix: usuability update from feedback * update documentation * docs: organize model-customization examples into serverless/serverful/deployment/evaluation subfolders * add job notification integ test * feat(telemetry): Add tracking for dry_run, notifications, reuse_resources, show_metrics, and stream_logs * update telemetry * update telemetry * fix: tail_lines in stream_logs returns last N events (true tail semantics) (#6164) - Add LogStreamer.poll_tail() for fetching last N events: - Stream mode (SMTJ): backward pagination via get_log_events nextBackwardToken - Filter mode (SMHP): filter_log_events with startFromHead=False, paginate until N matches collected (CW bounds pages by scan volume, not result count) - Multi-stream jobs: merge events across streams by timestamp, return globally last N - stream_log_loop: when tail_lines is set, call poll_tail() and return immediately - Refactor _stream_logs_smhp to delegate to LogStreamer + stream_log_loop, eliminating ~80 lines of duplicated inline polling logic - Validate start_time >= 2024-01-01 in _tail_filter_mode (CW API restriction) - Remove dead code: lines_printed counter no longer needed in SMHP forward loop - Add unit tests for poll_tail (stream mode, filter mode, multi-stream merge, multi-page pagination, pre-2024 validation) - Add unit tests for stream_log_loop tail_lines integration * Remove duplicated dry_run return * fix: unit tests * fix: integ test (dryrun, model reuse) * fix: model reuse for Nova model * fix: sagemaker-train test fix, remove extra dryrun tests * fix: 'ModelPackage' is not defined * fix: gpu integ test fixes * fix: key check before logging training job name (#6168) Co-authored-by: Syed Jafri <syedjfr@amazon.com> * Fix tests in trainers (#6169) Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com> * Master nova follow ups (#6170) * Fix tests in trainers * fix(iam): Remove iam:PassRole from caller validation to avoid false denials SimulatePrincipalPolicy without iam:PassedToService context value returns implicitDeny for condition-scoped PassRole policies like AmazonSageMakerFullAccess, causing false test failures. --------- Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com> --------- Co-authored-by: Ealynn Hsu <89547630+ehsu3@users.noreply.github.com> Co-authored-by: Ealynn Hsu <ealynnh@amazon.com> Co-authored-by: LN <133025223+amazeAmazing@users.noreply.github.com> Co-authored-by: Syed Jafri <syedjfr@amazon.com> Co-authored-by: Zhaoqi <jzhaoqwa@amazon.com> Co-authored-by: eliseharvey <108292155+eliseharvey@users.noreply.github.com> Co-authored-by: Elise Harvey <harveel@amazon.com> Co-authored-by: Sayem Kamal <sayemkamal12@gmail.com> Co-authored-by: sayemkam <sayemkam@amazon.com> Co-authored-by: papriwal <papriwal@amazon.com> Co-authored-by: rsareddy0329 <rsareddy0329@gmail.com> Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com>Syed Mujtaba · 04a71225 · 2026-08-09
- 9.2ETVMTRL Launch PR (#5919) * Add master-mtrl-trainer branch to PR checks * chore: update svc model (#2005) * feature: add RMP (Restricted Model Package) support for ModelBuilder (#2011) * Add master-mtrl-release branch to PR checks * feat: add RMP (Restricted Model Package) support for ModelBuilder - Add shared rmp_utils.py with is_restricted_model_package() and get_container_s3_uri() utilities - Fix _fetch_and_cache_recipe_config() crash when s3_uri is None - Fix _build_single_modelbuilder() non-LORA path to use model_package_name for RMP (CP resolves escrow server-side) - Fix _convert_model_data_source_to_local() to return None for RMP - Add unit tests and regression tests (14 passing) Builds on top of PR #2010 (shapes.py Optional fix + BedrockModelBuilder RMP). Rebase needed after #2010 merges. * fix: address PR review feedback - Rename rmp_utils.py to model_package_utils.py - Rename get_container_s3_uri to get_s3_uri_from_inference_spec with null checks - Add early RMP exit in _build_single_modelbuilder before containers[0] access - Add RMP guard in _deploy_model_customization and fetch_endpoint_names_for_base_model - Remove internal acronyms from comments and error messages * fix: simplify RMP detection, add Nova env vars, improve tests - Remove fallback detection — use managed_storage_type == Restricted only - Add Nova hosting config (image + env vars) for Nova RMP in build path - Conditionally pass image only when provided by user - Update unit tests: 22 tests covering all edge cases * fix: add enable_network_isolation for Nova restricted model packages --------- Co-authored-by: Gokul Anantha Narayanan <166456257+nargokul@users.noreply.github.com> Co-authored-by: Jonathan makunga <makung@amazon.com> * Merge RFT SDK changes from master-rft-sdk-integration to master-mtrl-release (#2015) * feat: Add sagemaker-rft SDK for AgenticRFT integration (#1977) * feat: Add sagemaker-rft subpackage for multi-turn RFT customer integration * fix: update sagemaker-rft for AgenticRFTRuntimeService integration * feat: Add aws_rft_sdk source package - RolloutFeedbackClient with SigV4-signed CompleteTrajectory and UpdateReward - @rft_handler decorator: auto-reports completion+reward on success, errors on failure - RFTContext using contextvars (not threading.local) for Strands thread compatibility - Strands wrap_model adapter: injects X-Rft-* headers via client_args default_headers - Maps both snake_case and camelCase metadata keys (jobId/rolloutId from TLM) * chore: Remove superseded aws-rft-sdk package The aws-rft-sdk/ directory was the original prototype, now fully superseded by sagemaker-rft/ which provides the same functionality under the sagemaker.rft namespace with proper packaging, Pydantic models, and additional adapters (LangChain). Keeping both causes confusion about which package to import. * chore: Add pre-built wheel for sagemaker-rft 0.1.0 Include distributable wheel and sdist so other teams can install directly without building from source. * Consolidate individual RFT headers into single X-RFT-Metadata header Replace three separate headers (X-Rft-Job-Arn, X-Trajectory-Id, X-Span-Id) with a single X-RFT-Metadata header containing a JSON object with: - job_id: training job ARN - experiment_id: groups turns into a single trajectory - rollout_id: unique ID for each rollout (replaces span-id) * Fix RolloutFeedbackClient field name mismatch with TLM metadata The TLM sends metadata with jobId/rolloutId but RolloutFeedbackClient expected job_arn/trajectory_id, causing complete_trajectory and update_reward to silently skip. Accept both naming conventions (snake_case and camelCase). * Fix inference params field name mismatch with TLM payload - decorators.py: Accept both inferenceParams (camelCase from TLM) and inference_params (snake_case) in payload - strands.py: Accept both maxTokens/topP (camelCase) and max_tokens/top_p (snake_case) when applying to model params * Fix SDK payload casing, rollout ID, and inference params injection 1. headers.py: Accept both camelCase (jobId, rolloutId, experimentId) and snake_case (job_arn, trajectory_id) from TLM payload. Use the passed rollout ID instead of generating a new UUID. 2. strands.py: Use model.update_config() instead of setting params dict directly — params is None on OpenAIModel, update_config is the supported API for dynamic parameter changes. 3. decorators.py: Already accepts both inferenceParams and inference_params (fixed earlier). * Fix update_config: wrap inference params in params={} for Strands API * Add status param to complete_trajectory and report_error/report_complete helpers - complete_trajectory() accepts status param ("ready" or "failed") - Added report_error() to mark trajectory as failed with optional reward - Added report_complete() convenience method for success path - 404 errors already handled gracefully via _signed_post exception handling * feat: Updated with correct headers for rft * feat: Added variable endpoint for rft runtime and feedback to complete trajectory and reward * feat: added temp auth.py module and updated feedback.py * feat: added temp auth.py module and updated feedback.py * Support list rewards in rft_handler and report_complete - rft_handler: when reward is a list, calls complete_trajectory() + update_reward(list) separately instead of report_complete(float). This supports multi-turn trajectories with per-turn rewards. - report_complete: accepts float | list[float]. * fix: sagemaker-rft SDK bearer token auth, auto lifecycle, region env var - feedback.py: Switch from SigV4 to bearer token auth via aws_sagemaker_token_generator.provide_token(). Add region fallback from AWS_REGION env var. Support both camelCase (jobId, rolloutId) and snake_case (job_arn, trajectory_id) metadata keys. Handle 404 gracefully. Add report_complete() and report_error() convenience methods. - decorators.py: rft_handler now auto-calls CompleteTrajectory + UpdateReward when result dict contains "reward" key. Auto-calls report_error on exceptions. Support inferenceParams (camelCase). - models.py: Region default reads from AWS_REGION env var. * fix: Handle error results and terminal trajectory status in SDK - decorators.py: _handle_result checks result["status"] == "error" and calls report_error() instead of report_complete(). Previously, agent returning {"status": "error", "reward": 0.0} was treated as success, calling CompleteTrajectory(status=ready) which conflicted with Runtime's failTrajectory(). Now errors are properly reported so TLM retries immediately instead of waiting 10 min timeout. - feedback.py: CompleteTrajectory and UpdateReward gracefully handle 400 "not in valid status" (trajectory already failed by Runtime). Logs warning and skips instead of raising exception. * fix(rft): make feedback reporting non-fatal and fix endpoint/timeout - Move _handle_result to try/except in else clause so feedback failures do not prevent returning the rollout result - Fix _build_endpoint to not include "prod" in the URL prefix - Increase feedback HTTP timeout from 30s to 120s * Handle trajectory-already-processed errors in rft_handler decorator Centralize detection of 'not in valid status' and 'Cannot transition trajectory' errors into a shared helper. The decorator now catches these from the agent function and returns {status: skipped} instead of re-raising, so agents no longer need per-project workarounds. * fix: use Optional[] syntax in Pydantic models for Python 3.9 compatibility * refactor(rft): use sagemaker-core token generator instead of inline implementation Replace the inline SigV4 token signing logic and aws-sagemaker-token-generator fallback with sagemaker.core.token_generator.generate_token from sagemaker-core. Add sagemaker-core as a dependency in pyproject.toml. * refactor: move rft module from sagemaker-rft to sagemaker-train Move the rft subpackage from the standalone sagemaker-rft package into sagemaker-train as sagemaker.train.rft. Update all internal imports accordingly. Add requests and pydantic to sagemaker-train dependencies. Remove the sagemaker-rft package directory. * refactor(rft): remove auth.py wrapper, use generate_token directly The get_rft_api_key wrapper in auth.py was just passing args through to generate_token. Call generate_token directly in feedback.py instead. --------- Co-authored-by: Tritin Truong <tttritin@amazon.com> Co-authored-by: Barret Pickett <mrpic@amazon.com> Co-authored-by: James Yu <jamesfyu@amazon.com> * refactor: Rename headers, URIs, and decorator (#1990) * Rename finetuning-job-runtime endpoint to job-runtime (#1999) * Update strands.py (#2017) --------- Co-authored-by: Mike Shen <109769013+xiaoxshe@users.noreply.github.com> Co-authored-by: Tritin Truong <tttritin@amazon.com> Co-authored-by: Barret Pickett <mrpic@amazon.com> Co-authored-by: James Yu <jamesfyu@amazon.com> * feat: support Restricted Model Package (RMP) deployment to Bedrock OD (#2012) MTRL training outputs checkpoints to Restricted Model Packages where S3 URIs are hidden (ManagedStorageType: Restricted). This change: 1. sagemaker-core: Make s3_uri, s3_data_type, compression_type optional in S3ModelDataSource so ModelPackage.get() can deserialize RMPs without crashing on missing s3_uri field. 2. sagemaker-serve: BedrockModelBuilder now uses customModelDataSource.modelPackageArnDataSource when the model artifact is a model package ARN (RMP), instead of the unsupported modelSourceConfig.s3DataSource path. Falls back to model package ARN in _get_s3_artifacts when s3_uri is hidden for Nova RMP models. Tested end-to-end: ModelPackage.get() on RMP -> BedrockModelBuilder.deploy() -> Bedrock OD endpoint Active -> inference invocation successful. Co-authored-by: Mahima Chaudhary <mahchy@amazon.com> * Master mtrl trainer (#2018) * feat: Add MultiTurnRLTrainer for Agentic RFT jobs (#1988) * MTRL Evaluator (#1989) * feat: Add SageMaker token generator to sagemaker-core (#1983) * Feature processor v3 (#5565) * Feature store v3 (#5490) * feat: Add Feature Store Support to V3 * Add feature store tests --------- Co-authored-by: adishaa <adishaa@amazon.com> * feat: feature_processor v3 * integ tests * fix * chore(docs): Add API docs * fix: Fix flaky integ tests * fix diff * chore: rename parameter + cleanup comments * Feature store v3 (#5490) * feat: Add Feature Store Support to V3 * Add feature store tests --------- Co-authored-by: adishaa <adishaa@amazon.com> * add pyspark to test deps * add test deps * fix unit test deps * pin setuptools<82 for feature-processor and unit tests * Set JAVA_HOME for integ tests which requires java * fix spark session bug * fix(feature-processor): Fix Spark session config and Ivy cache race condition Isolate Ivy cache per Spark session via spark.jars.ivy to prevent concurrent pytest-xdist workers from corrupting shared /root/.ivy2/cache during Maven dependency resolution in CI. * revert previous change + create different ivy cache per test to fix concurrent writes in CI * revert changes to sagemaker-core * refactor(feature-processor): Migrate to FeatureGroup resource API - Replace sagemaker_session.describe_feature_group() calls with FeatureGroup.get() - Update _input_loader.py to use FeatureGroup resource attributes instead of dictionary access - Update feature_scheduler.py to use FeatureGroup.get() and access creation_time as attribute - Update _feature_group_lineage_entity_handler.py to return FeatureGroup resource instead of Dict - Remove unused imports (Dict, Any, FEATURE_GROUP, CREATION_TIME constants) - Replace dictionary key access with typed resource properties (offline_store_config, data_catalog_config, event_time_feature_name, etc.) - Update unit tests to reflect new FeatureGroup resource API usage - Improves type safety and reduces reliance on dictionary-based API responses * add `build` to test_requirements * add upper bounds for test dependencies * move feature-processor config to sagemaker-mlops optional deps --------- Co-authored-by: Aditi Sharma <165942273+Aditi2424@users.noreply.github.com> Co-authored-by: adishaa <adishaa@amazon.com> Co-authored-by: Basssem Halim <bhhalim@amazon.com> Co-authored-by: Molly He <mollyhe@amazon.com> * Added iso regions to dji-lmi (#5595) * Add docker-compose path to allow local training (#5598) * Add docker-compose path * Check for MacOS * Remove Unused method (#5593) * V3 Bug Fixes (#5601) * V3 Bug Fixes * fix(model_builder): Only set s3_upload_path for S3 URIs in passthrough In _build_for_passthrough(), model_path could be a local /tmp path. Setting s3_upload_path to a local path caused CreateModel API to reject the modelDataUrl with a validation error since it requires s3:// or https:// URIs. Now only S3 URIs are assigned to s3_upload_path; local paths are handled separately by _prepare_for_mode() in LOCAL_CONTAINER mode. * Test fixes * Bug fix 3 and 4 * fix: Add PipelineVariable support to ModelTrainer fields (fixes #5524) (#5608) * fix: Add PipelineVariable support to ModelTrainer fields (fixes #5524) Extend StrPipeVar type to ModelTrainer's direct fields: - training_image: Optional[str] -> Optional[StrPipeVar] - algorithm_name: Optional[str] -> Optional[StrPipeVar] - training_input_mode: Optional[str] -> Optional[StrPipeVar] - environment: Dict[str, str] -> Dict[str, StrPipeVar] This follows the existing V3 pattern already used by SourceCode, OutputDataConfig, and Compute (for instance_type). The StrPipeVar type alias and PipelineVariable.__get_pydantic_core_schema__() already exist in the codebase. This unblocks V2->V3 migration for SageMaker Pipelines users who need to pass ParameterString to ModelTrainer fields. Fixes #5524 * test: Add unit tests for PipelineVariable support + fix PipelineVariable-safe logging - Add test_model_trainer_pipeline_variable.py with 9 tests: - 4 PipelineVariable acceptance tests (training_image, algorithm_name, training_input_mode, environment) - 4 regression tests (real string values still work) - 1 invalid type rejection test - Fix PipelineVariable-safe logging in model_post_init (avoid __str__ on PipelineVariable which raises TypeError) All 57 tests pass (48 existing + 9 new, 0 regressions). --------- Co-authored-by: Amit Modi <modiamit@amazon.com> * Fix model registration with a model card (#5611) * Add docker-compose path * Check for MacOS * Fix model registration with a model card * Account for both ModelCard and ModelPackageModelCard objects * Add unit tests for model card during model registration * updated the SDK to use latest LMI image for sdk v3.x (#5616) * add EUCS to Jumpstart region config (#5615) Co-authored-by: Molly He <mollyhe@amazon.com> * Fix handling of training step dependencies to allow successful pipeline creation (#5618) * Add docker-compose path * Check for MacOS * Fix model registration with a model card * Account for both ModelCard and ModelPackageModelCard objects * Add unit tests for model card during model registration * Fix handling of dependencies in get_training_code_hash workflow utility * Update docstring * Add unit tests * sagemaker-core rich upper bound relax back to 15.0.0 (#5620) * Release sagemaker-core 2.5.1 (#5623) * Update changelog for sagemaker-core 2.5.1 (#5624) * Release sagemaker-core 2.5.1 * Update changelog for sagemaker-core 2.5.1 * docs: Add migration tool (MCP server) section to migration guide (#5628) * docs: Add migration tool (MCP server) section to migration guide Add instructions for installing and configuring the SageMaker SDK migration MCP server tool. Includes setup for Kiro, Kiro CLI, VS Code (Cline), Claude Desktop, and Cursor. Documents available tools (analyze_code, transform_code, validate_code, ask_question), example usage, and troubleshooting steps. * docs: Update Feature Store status to supported in migration guide Feature Store is now supported in V3 via sagemaker.core.resources.FeatureGroup and FeatureStore. Update the status from REMOVED to SUPPORTED. * fix: resolve PermissionError during local mode cleanup of root-owned Docker files (#5629) * fix: use docker fallback to clean up root-owned files in local mode * Remove alpine * Use network flag * Add -mindepth * Use chmod -R 777 via Docker * Add unit test to sagemaker-core for permissionError docker fix * Migration guide update (#5633) * docs: Add migration tool (MCP server) section to migration guide Add instructions for installing and configuring the SageMaker SDK migration MCP server tool. Includes setup for Kiro, Kiro CLI, VS Code (Cline), Claude Desktop, and Cursor. Documents available tools (analyze_code, transform_code, validate_code, ask_question), example usage, and troubleshooting steps. * docs: Update Feature Store status to supported in migration guide Feature Store is now supported in V3 via sagemaker.core.resources.FeatureGroup and FeatureStore. Update the status from REMOVED to SUPPORTED. * docs(migration): Add Codex CLI, VS Code Copilot, and Roo Code to MCP server IDE setup table Add configuration locations for additional IDEs that support the SageMaker migration MCP server: VS Code with Copilot, VS Code with Roo Code extension, and Codex CLI. * Migration guide update (#5636) * docs: Add migration tool (MCP server) section to migration guide Add instructions for installing and configuring the SageMaker SDK migration MCP server tool. Includes setup for Kiro, Kiro CLI, VS Code (Cline), Claude Desktop, and Cursor. Documents available tools (analyze_code, transform_code, validate_code, ask_question), example usage, and troubleshooting steps. * docs: Update Feature Store status to supported in migration guide Feature Store is now supported in V3 via sagemaker.core.resources.FeatureGroup and FeatureStore. Update the status from REMOVED to SUPPORTED. * docs(migration): Add Codex CLI, VS Code Copilot, and Roo Code to MCP server IDE setup table Add configuration locations for additional IDEs that support the SageMaker migration MCP server: VS Code with Copilot, VS Code with Roo Code extension, and Codex CLI. * docs: Update MCP server name from sagemaker-migration-mcp to sagemaker-sdk-helper Replace all references to the deprecated sagemaker-migration-mcp binary with the correct sagemaker-sdk-helper command and server name across installation, configuration, and troubleshooting sections. * fix(tuner): Include sm_drivers channel in HyperparameterTuner jobs (#5634) * fix(tuner): Include sm_drivers channel in HyperparameterTuner jobs When ModelTrainer has distributed=Torchrun(), the sm_drivers channel contains torchrun_driver.py and sm_train.sh which are required for multi-GPU execution. The tuner was not building this channel, causing the framework container to fall back to the legacy single-GPU entry point (python train.py) instead of torchrun. This caused a tensor size mismatch (batch_size vs accumulated_batch) in TRL's compute_loss when gradient_accumulation_steps > 1, because the single-process path doesn't partition batches across ranks. Fix: Replace _upload_source_code_and_configure_hyperparameters with _build_driver_and_code_channels that replicates ModelTrainer's channel building logic (sm_drivers, code, distributed.json, sourcecode.json, sm_train.sh). Also pass through environment and VPC config. * fix(tuner): Harden _build_training_job_definition against missing attributes - Use getattr with fallback for static_hyperparameters (fixes test_build_training_job_definition_includes_internal_channels) - Guard _prepare_model_trainer_for_tuning with isinstance check on entry_script to avoid calling _build_driver_and_code_channels on MagicMock model trainers - Guard environment passthrough with isinstance(env, dict) check - Guard VPC config passthrough with try/except for mock safety * fix(test): Rewrite tuner distributed integ test to match CI patterns - Use sagemaker_session fixture from conftest (auto-resolves role/region) - Use ml.m5.xlarge CPU instance (cheaper, available in CI) - Remove hardcoded role ARN and training_mode - Remove @pytest.mark.slow (not registered in CI config) - Use module-level function instead of class (matches other integ tests) - Use DEFAULT_CPU_IMAGE consistent with test_model_trainer.py * fix(tuner): Upload sourcedir.tar.gz for framework container compatibility The HPT API uses the legacy framework container path which expects sagemaker_submit_directory (a tar.gz on S3) to be downloaded and extracted to /opt/ml/code/. The previous approach of using a 'code' input channel mounted the code at /opt/ml/input/data/code/ instead, causing 'No such file or directory' errors. Fix: Create and upload sourcedir.tar.gz to S3, set both sagemaker_program and sagemaker_submit_directory hyperparameters. Remove the separate 'code' input channel since the framework container handles code extraction via sagemaker_submit_directory. * test(tuner): Add unit tests for driver/code channel building Add 25 unit tests covering the tuner changes from PR #5634: - _prepare_model_trainer_for_tuning guard logic - _build_driver_and_code_channels sm_drivers channel creation - _build_training_job_definition _tuner_channels inclusion - Environment and VPC config passthrough - sourcedir.tar.gz upload and sagemaker_submit_directory HP - static_hyperparameters getattr fallback * feat(ci): Add Fortress Code Reviewer security scan workflow (#5639) Add GitHub Actions workflow to run Fortress Code Reviewer security scan on every PR against the master branch. The workflow: - Triggers on pull_request_target against master - Performs collaborator check (auto-approve for collaborators, manual approval for external contributors) - Configures AWS credentials via OIDC - Triggers the sagemaker-python-sdk-ci-fortress-scan CodeBuild project The CodeBuild project installs Fortress at runtime from S3-hosted wheels and uses Bedrock (Claude) to analyze code for security vulnerabilities. --- X-AI-Prompt: Add Fortress security scan GitHub workflow for PR scanning X-AI-Tool: Kiro * fixes for model builder (#5631) * fixes for model builder * add nova model support * fix env_vars merge, update integ test for LORA two-step deployment, fix unit tests for nova model support - env_vars: append recipe/nova config to existing env_vars instead of skipping - integ test: verify both base IC and adapter IC creation for LORA models - unit tests: add _is_nova_model mock to accommodate nova model support changes * update codegen to mark MinMemoryRequiredInMb as optional DescribeInferenceComponent returns empty ComputeResourceRequirements for adapter ICs (created with BaseInferenceComponentName), but the service model still marks MinMemoryRequiredInMb as required. Add a REQUIRED_TO_OPTIONAL_OVERRIDES config in the codegen so re-running shapes generation produces the correct Optional field. * add retry for adapter IC creation on transient endpoint-not-found * model builder fixes * Skip test_deploy_from_training_job: parallel cleanup race condition under investigation --------- Co-authored-by: Joshua Towner <jjtowner@amazon.com> Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com> * Bedrock fix (#5642) * fix(bedrock): Poll for model Active status before creating deployment Add _wait_for_model_active() to poll get_custom_model until the model reaches Active status before calling create_custom_model_deployment. This fixes ValidationException when the custom model is not yet ready for deployment after create_custom_model returns. * feat(bedrock): Harden BedrockModelBuilder for production readiness Extract _is_nova_model() helper to eliminate duplicated Nova detection logic across deploy() and _get_s3_artifacts(). Uses getattr with safe defaults instead of fragile hasattr chains. Add input validation to deploy() and create_deployment(): - Raise ValueError when model_package is not set - Raise ValueError when custom_model_name or role_arn missing for Nova deployments - Raise ValueError when model_arn is empty in create_deployment Move json and urlparse imports to module level (were previously imported inside _get_checkpoint_uri_from_manifest). Replace f-string logging with lazy %s formatting throughout. Initialize status=None before the polling loop in _wait_for_model_active to avoid UnboundLocalError if the loop body never executes. Rewrite unit tests (43 tests) with full coverage: - _is_nova_model: recipe_name, hub_content_name, case insensitivity, missing base_model, None fields - __init__: None model, TrainingJob, ModelPackage - Client singletons: caching, injection - _fetch_model_package: ModelPackage, TrainingJob, ModelTrainer, unknown type - _get_s3_artifacts: None package, non-Nova, Nova delegation, Nova fallback - _get_checkpoint_uri_from_manifest: success, missing key, NoSuchKey, not TrainingJob, no artifacts, invalid JSON - _wait_for_model_active: immediate, polling, Failed, timeout - create_deployment: polling chain, extra kwargs, empty/None ARN - deploy: non-Nova, Nova full chain, hub_content_name detection, default deployment name, tags, missing params, None stripping Add integration tests for Nova E2E deployment: - Training job existence and status verification - Builder creation and Nova detection via _is_nova_model - S3 artifacts checkpoint validation - Full deploy-with-polling flow (marked @pytest.mark.slow) - Timeout behavior on bogus ARN - Validation error paths (no model_package, empty model_arn) - Resource cleanup fixture for deployments and custom models * feat(bedrock): Add deployment status polling after CreateCustomModelDeployment Previously create_deployment() only polled for the custom model to reach Active status before calling CreateCustomModelDeployment, but did not wait for the deployment itself to become Active. This caused callers to receive a deployment ARN that was still in Creating state, requiring manual polling in user code. Add _wait_for_deployment_active() that polls get_custom_model_deployment until status reaches Active, raises RuntimeError on Failed, and times out after max_wait seconds (default 3600s, poll interval 30s). Wire it into create_deployment() so the full flow is now: 1. _wait_for_model_active (poll model creation) 2. create_custom_model_deployment (API call) 3. _wait_for_deployment_active (poll deployment creation) Gracefully skips deployment polling if the API response does not contain a customModelDeploymentArn. Unit tests (48 passing): - _wait_for_deployment_active: immediate Active, polling, Failed status, timeout - create_deployment: full model+deployment polling chain, skip polling when no ARN in response - deploy Nova chain: updated to verify deployment polling * fix(integ): Fix region handling and add get-or-create Nova training job The TestModelCustomizationDeployment integ tests were failing with DescribeTrainingJob 'Requested resource not found' because the SageMaker SDK caches the first session's region internally. The session-scoped cleanup_e2e_endpoints fixture (autouse) was creating a session in us-east-1 (default) before the class fixtures could set us-west-2, causing all subsequent TrainingJob.get calls to hit the wrong region. Fix by setting AWS_DEFAULT_REGION=us-west-2 in the cleanup_e2e_endpoints fixture before any SageMaker session is created. Add tests/integ/conftest.py with a session-scoped nova_training_job_name fixture that implements get-or-create: - Checks if sdk-integ-nova-micro-sft exists and is Completed - If InProgress, waits for completion - If not found, uploads minimal training data to S3 and launches a Nova Micro SFT training job via SFTTrainer - Reused across test_bedrock_nova_e2e.py and TestBedrockNovaDeployment in test_model_customization_deployment Update both Nova test files to use the shared fixture instead of hardcoded training job names. * fix(integ): Use SAGEMAKER_REGION for cross-region training job lookup The SageMaker SDK's SageMakerClient reads SAGEMAKER_REGION env var at init time and caches the region for all subsequent API calls. The cleanup_e2e_endpoints session fixture was the first to create a SageMakerClient (in the default region), which then poisoned all subsequent TrainingJob.get calls regardless of the region parameter. Fix by setting SAGEMAKER_REGION=us-west-2 in cleanup_e2e_endpoints before any SDK session is created, since all resources in this test file live in us-west-2. The env var is restored after cleanup. In CodeBuild (us-west-2) this is a no-op since the default region already matches. The other test files (triton, tei, tgi) are not affected since they have their own fixtures and don't import from this file. * refactor(integ): Replace Nova integ tests with example notebook Remove us-east-1 Nova integration tests that cannot run in the us-west-2 CodeBuild environment: - Delete tests/integ/test_bedrock_nova_e2e.py - Delete tests/integ/conftest.py (Nova get-or-create fixture) - Remove TestBedrockNovaDeployment class from test_model_customization_deployment.py Add example_notebooks/bedrock_nova_deployment.ipynb covering the full Nova workflow: SFTTrainer fine-tuning, BedrockModelBuilder deploy with model+deployment polling, inference, and cleanup. The BedrockModelBuilder source code and unit tests (48 passing) are unchanged. The us-west-2 integ tests for non-Nova Bedrock deployment (TestModelCustomizationDeployment) remain. * docs: Add Bedrock model builder example notebooks Add notebooks demonstrating Bedrock deployment workflows: - bedrock-modelbuilder-deployment-nova.ipynb: Nova model deployment via BedrockModelBuilder with SFTTrainer fine-tuning - boto3_deployment_notebook.ipynb: Direct boto3 Bedrock deployment - model_builder_deployment_notebook(1).ipynb: ModelBuilder deployment - 07-ml-model-development(1).ipynb: ML model development workflow - sagemaker-serve/example_notebooks/bedrock_nova_deployment.ipynb: Clean Nova deployment example with polling, inference, and cleanup * docs: Add Bedrock model builder example notebooks Add notebooks demonstrating Bedrock deployment workflows: - bedrock-modelbuilder-deployment-nova.ipynb: Nova model deployment via BedrockModelBuilder with SFTTrainer fine-tuning - boto3_deployment_notebook.ipynb: Direct boto3 Bedrock deployment - model_builder_deployment_notebook(1).ipynb: ModelBuilder deployment - 07-ml-model-development(1).ipynb: ML model development workflow - sagemaker-serve/example_notebooks/bedrock_nova_deployment.ipynb: Clean Nova deployment example with polling, inference, and cleanup * fix(serve): Update Nova Bedrock deployment notebook with working e2e flow Simplify notebook to use existing completed training job with BedrockModelBuilder deploy flow. Fix Nova inference content format to use array of {text: ...} objects. Remove broken SFTTrainer cells that fail due to botocore service model mismatch. * Update CHANGELOG 3.6.0 (#5649) * Update CHANGELOG.md sagemaker-core * Update VERSION sagemaker-core * Update CHANGELOG.md sagemaker-train * Update VERSION sagemaker-train * Update pyproject.toml sagemaker-train * Update CHANGELOG.md sagemaker-serve * Update VERSION sagemaker-serve * Update pyproject.toml sagemaker-serve * Update CHANGELOG.md sagemaker-mlops * Update VERSION sagemaker-mlops * Update pyproject.toml sagemaker-mlops * Update VERSION meta * Update CHANGELOG.md meta * Update pyproject.toml meta * Eval Support Update (#5658) * fix(evaluate): Remove GPT OSS model evaluation restriction Remove the check that blocked evaluation for openai-reasoning-gpt-oss-20b and openai-reasoning-gpt-oss-120b base models. * test(evaluate): Update GPT OSS tests to verify models are allowed Update TestGPTOSSModelValidation to assert that openai-reasoning-gpt-oss-20b and openai-reasoning-gpt-oss-120b models can be used for evaluation, matching the removal of the restriction in base_evaluator. * feature: Add Support for AWS Batch Quota Management Job Submission and Job Priority Update (#5659) * feature: [SDKv3]Add Support for QM Job Submission and Job Priority Update (#1970) * Trigger checks in changed modules and dependent modules (#1958) * Update pr workflow (#1963) * Trigger checks in changed modules and dependent modules * Removing github token dependency * Add back GH_PAT token to detect changes (#1965) * feature: Add Support for QM Job Submission and Job Priority Update --------- Co-authored-by: aviruthen <91846056+aviruthen@users.noreply.github.com> * feature: Updating aws_batch TrainingQueue integration test to support quota management. (#1978) * feature: Added an example notebook for QuotaManagement job submission on AWS Batch TrainingQueues. (#1980) * fix: aws_batch/test_training_queue QM unit test fix --------- Co-authored-by: mnganesh-amzn <mnganesh@amazon.com> Co-authored-by: aviruthen <91846056+aviruthen@users.noreply.github.com> * Migration MD Update (#5655) * updated the SDK to use latest LMIv22 image for sdk v3.x (#5640) * fix: Sync Nova hosting configs with AGISageMakerInference (#5664) Align _NOVA_HOSTING_CONFIGS CONTEXT_LENGTH and MAX_CONCURRENCY values with ALLOWLISTED_CONFIGURATIONS from AGISageMakerInference constants.py. Key changes: - micro: correct context/concurrency for g5, g6 instances; add g6e types - lite: add g6.12xlarge, g6.24xlarge; fix p5 to 128000 context - pro: remove unsupported g6.48xlarge; fix p5 to 24000/1 - lite-v2: add g6.48xlarge; fix p5 to 128000 context * feat: MLflow metrics visualization, enhanced wait UI, and eval job links (#5662) * Intermediary checkpoint * Evaluation job update * Fix studio domain mismatch for url, update text color, add link of evaluation job * Add underscore to fine-tune and eval job links * Update link to console, conditionally display studio link, update link color to blue * Always show console link, conditional show studio link * Minor update to execution link names * Fix region issue for studio url * Revert notebook change to original * Address PR readiness * Fix sagemaker-train unit tst * Update resources_codegen based on sagemaker-core change * feature: add telemetry attribution module for SDK usage provenance (#5661) * feature: add telemetry attribution module for SDK usage provenance * feature: add TrainingJob ARN to telemetry for training jobs and fixed bug with telemetry not being sent for *Trainer.train() if sagemaker_session is not provided * adding createdBy metadata to user agent string if attribution env var has been set to aid in resource attribution * fix: removed unused patch on builtins.open in test_create_with_byoc which was not being used and causing unintended patches to open calls elsewhere --------- Co-authored-by: Ryan Tanaka <rrtanaka@amazon.com> Co-authored-by: Molly He <mollyhe@amazon.com> * feature: extend list_jobs_by_share for quota_share_name (#5669) Co-authored-by: houtampl <houtampl@amazon.com> * fix: aws_batch integ test resources are now uniquely named by test run. (#5666) * Support IAM role for BaseEvaluator (#5671) * Updating changelog, version, and pyproject files (#5673) * fix(evaluate): Remove ModelPackageConfig from EvaluateBaseModel steps (#5635) When evaluate_base_model=True, the EvaluateBaseModel step in both DETERMINISTIC_TEMPLATE and CUSTOM_SCORER_TEMPLATE incorrectly included ModelPackageConfig with SourceModelPackageArn, causing the base model evaluation to load fine-tuned model weights instead of using only the base model from the public hub. This made both evaluations identical, leading users to believe fine-tuning had no effect. Remove ModelPackageConfig from the EvaluateBaseModel step in both templates so it only uses BaseModelArn from ServerlessJobConfig. The EvaluateCustomModel step retains ModelPackageConfig to correctly load fine-tuned weights. This is consistent with the fix already applied to the LLMAJ_TEMPLATE. --- X-AI-Prompt: Fix BenchMarkEvaluator evaluate_base_model bug from D406780217 X-AI-Tool: Kiro sim: https://t.corp.amazon.com/D406780217 * Fix: hardcode handler_name = "lambda_function.lambda_handler" to match the zip entry name. (#5692) * Fix lambda function handler name * Add integ test * Update integ test to wait for lambda call * feat: add telemetry emitter to ScriptProcessor and FrameworkProcessor run methods (#5697) Co-authored-by: Ryan Tanaka <rrtanaka@amazon.com> * fix: respect accept_eula in ModelBuilder LoRA deployment path (#5705) * Update accept_eula to respect user setup * Enable EULA acceptance in model customization tests Set accept_eula to True in model builder to fix tests * fix: add missing model_path attr in TestLoraAcceptEula To fix failing unit tests in PR: https://github.com/aws/sagemaker-python-sdk/pull/5696 * fix(tests): fix TestLoraAcceptEula missing dataclass attrs and patches --------- Co-authored-by: Molly He <mollyhe@amazon.com> Co-authored-by: Gokul Anantha Narayanan <166456257+nargokul@users.noreply.github.com> * chore: Updated changelog, version and pyproject.toml for release (#5706) * feat: Add SageMaker token generator to sagemaker-core Embed the aws-sagemaker-token-generator library into sagemaker.core so users can generate SageMaker bearer tokens without installing a separate wheel. Usage: from sagemaker.core.aws_sagemaker_token_generator import provide_token token = provide_token(region='us-east-1') --------- Co-authored-by: Bassem Halim <bassemamir459@gmail.com> Co-authored-by: Aditi Sharma <165942273+Aditi2424@users.noreply.github.com> Co-authored-by: adishaa <adishaa@amazon.com> Co-authored-by: Basssem Halim <bhhalim@amazon.com> Co-authored-by: Molly He <mollyhe@amazon.com> Co-authored-by: Zachary David Saunders <zsaund@amazon.com> Co-authored-by: Bobby Lindsey <bobbywlindsey@users.noreply.github.com> Co-authored-by: Gokul Anantha Narayanan <166456257+nargokul@users.noreply.github.com> Co-authored-by: Amit <modi.osu@gmail.com> Co-authored-by: Amit Modi <modiamit@amazon.com> Co-authored-by: Rohit Kumar Srivastava <141.srivastava@gmail.com> Co-authored-by: IshaChid76 <49986634+IshaChid76@users.noreply.github.com> Co-authored-by: jam-jee <jamjee@amazon.com> Co-authored-by: rsareddy0329 <rsareddy0329@gmail.com> Co-authored-by: Joshua Towner <jjtowner@amazon.com> Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com> Co-authored-by: David Lindskog <davlind@amazon.com> Co-authored-by: mnganesh-amzn <mnganesh@amazon.com> Co-authored-by: aviruthen <91846056+aviruthen@users.noreply.github.com> Co-authored-by: Ryan <ryantanaka.y@gmail.com> Co-authored-by: Ryan Tanaka <rrtanaka@amazon.com> Co-authored-by: ampleh <22372465+ampleh@users.noreply.github.com> Co-authored-by: houtampl <houtampl@amazon.com> Co-authored-by: Syed Mujtaba <42322958+mujtaba1747@users.noreply.github.com> * feat: Add sagemaker-rft SDK for AgenticRFT integration (#1977) * feat: Add sagemaker-rft subpackage for multi-turn RFT customer integration * fix: update sagemaker-rft for AgenticRFTRuntimeService integration * feat: Add aws_rft_sdk source package - RolloutFeedbackClient with SigV4-signed CompleteTrajectory and UpdateReward - @rft_handler decorator: auto-reports completion+reward on success, errors on failure - RFTContext using contextvars (not threading.local) for Strands thread compatibility - Strands wrap_model adapter: injects X-Rft-* headers via client_args default_headers - Maps both snake_case and camelCase metadata keys (jobId/rolloutId from TLM) * chore: Remove superseded aws-rft-sdk package The aws-rft-sdk/ directory was the original prototype, now fully superseded by sagemaker-rft/ which provides the same functionality under the sagemaker.rft namespace with proper packaging, Pydantic models, and additional adapters (LangChain). Keeping both causes confusion about which package to import. * chore: Add pre-built wheel for sagemaker-rft 0.1.0 Include distributable wheel and sdist so other teams can install directly without building from source. * Consolidate individual RFT headers into single X-RFT-Metadata header Replace three separate headers (X-Rft-Job-Arn, X-Trajectory-Id, X-Span-Id) with a single X-RFT-Metadata header containing a JSON object with: - job_id: training job ARN - experiment_id: groups turns into a single trajectory - rollout_id: unique ID for each rollout (replaces span-id) * Fix RolloutFeedbackClient field name mismatch with TLM metadata The TLM sends metadata with jobId/rolloutId but RolloutFeedbackClient expected job_arn/trajectory_id, causing complete_trajectory and update_reward to silently skip. Accept both naming conventions (snake_case and camelCase). * Fix inference params field name mismatch with TLM payload - decorators.py: Accept both inferenceParams (camelCase from TLM) and inference_params (snake_case) in payload - strands.py: Accept both maxTokens/topP (camelCase) and max_tokens/top_p (snake_case) when applying to model params * Fix SDK payload casing, rollout ID, and inference params injection 1. headers.py: Accept both camelCase (jobId, rolloutId, experimentId) and snake_case (job_arn, trajectory_id) from TLM payload. Use the passed rollout ID instead of generating a new UUID. 2. strands.py: Use model.update_config() instead of setting params dict directly — params is None on OpenAIModel, update_config is the supported API for dynamic parameter changes. 3. decorators.py: Already accepts both inferenceParams and inference_params (fixed earlier). * Fix update_config: wrap inference params in params={} for Strands API * Add status param to complete_trajectory and report_error/report_complete helpers - complete_trajectory() accepts status param ("ready" or "failed") - Added report_error() to mark trajectory as failed with optional reward - Added report_complete() convenience method for success path - 404 errors already handled gracefully via _signed_post exception handling * feat: Updated with correct headers for rft * feat: Added variable endpoint for rft runtime and feedback to complete trajectory and reward * feat: added temp auth.py module and updated feedback.py * feat: added temp auth.py module and updated feedback.py * Support list rewards in rft_handler and report_complete - rft_handler: when reward is a list, calls complete_trajectory() + update_reward(list) separately instead of report_complete(float). This supports multi-turn trajectories with per-turn rewards. - report_complete: accepts float | list[float]. * fix: sagemaker-rft SDK bearer token auth, auto lifecycle, region env var - feedback.py: Switch from SigV4 to bearer token auth via aws_sagemaker_token_generator.provide_token(). Add region fallback from AWS_REGION env var. Support both camelCase (jobId, rolloutId) and snake_case (job_arn, trajectory_id) metadata keys. Handle 404 gracefully. Add report_complete() and report_error() convenience methods. - decorators.py: rft_handler now auto-calls CompleteTrajectory + UpdateReward when result dict contains "reward" key. Auto-calls report_error on exceptions. Support inferenceParams (camelCase). - models.py: Region default reads from AWS_REGION env var. * fix: Handle error results and terminal trajectory status in SDK - decorators.py: _handle_result checks result["status"] == "error" and calls report_error() instead of report_complete(). Previously, agent returning {"status": "error", "reward": 0.0} was treated as success, calling CompleteTrajectory(status=ready) which conflicted with Runtime's failTrajectory(). Now errors are properly reported so TLM retries immediately instead of waiting 10 min timeout. - feedback.py: CompleteTrajectory and UpdateReward gracefully handle 400 "not in valid status" (trajectory already failed by Runtime). Logs warning and skips instead of raising exception. * fix(rft): make feedback reporting non-fatal and fix endpoint/timeout - Move _handle_result to try/except in else clause so feedback failures do not prevent returning the rollout result - Fix _build_endpoint to not include "prod" in the URL prefix - Increase feedback HTTP timeout from 30s to 120s * Handle trajectory-already-processed errors in rft_handler decorator Centralize detection of 'not in valid status' and 'Cannot transition trajectory' errors into a shared helper. The decorator now catches these from the agent function and returns {status: skipped} instead of re-raising, so agents no longer need per-project workarounds. * fix: use Optional[] syntax in Pydantic models for Python 3.9 compatibility * refactor(rft): use sagemaker-core token generator instead of inline implementation Replace the inline SigV4 token signing logic and aws-sagemaker-token-generator fallback with sagemaker.core.token_generator.generate_token from sagemaker-core. Add sagemaker-core as a dependency in pyproject.toml. * refactor: move rft module from sagemaker-rft to sagemaker-train Move the rft subpackage from the standalone sagemaker-rft package into sagemaker-train as sagemaker.train.rft. Update all internal imports accordingly. Add requests and pydantic to sagemaker-train dependencies. Remove the sagemaker-rft package directory. * refactor(rft): remove auth.py wrapper, use generate_token directly The get_rft_api_key wrapper in auth.py was just passing args through to generate_token. Call generate_token directly in feedback.py instead. --------- Co-authored-by: Tritin Truong <tttritin@amazon.com> Co-authored-by: Barret Pickett <mrpic@amazon.com> Co-authored-by: James Yu <jamesfyu@amazon.com> * MTRL Evaluator * fixes * Integ test changes * fix: Remove unimplemented trajectory stub from MTRL evaluator Remove _MTRLTurn, _MTRLTrajectory classes and _fetch_mtrl_trajectory function that raised NotImplementedError. These were stubbed for a future phase and should not ship to customers. Also remove the get_trajectory reference from the evaluate() docstring. * refactor: Align MTRL evaluator with standard evaluator UX contract - Type evaluate() return as MTRLEvaluationExecution - Replace custom _start_execution_boto3 with shared _start_execution path (adds pipeline tagging for discovery via get_all) - Remove _find_existing_pipeline (handled by shared infrastructure) - Remove custom wait()/refresh()/_print_trailing_logs() overrides from MTRLEvaluationExecution (uses sagemaker-core based parent methods) - Remove _get_results() and _show_mtrl_results (show_results removed) - Add proper get_all classmethod with telemetry and yield semantics - Add TYPE_CHECKING import for return type annotation * feat: Add 3P agent (Lambda) integration test for MTRL evaluator - Add test_mtrl_evaluator_3p_agent.py with 4 test cases covering Lambda ARN string, AgentLambda object, wait-for-completion, and get_all discoverability - Fix _resolve_agent_arn to handle AgentLambda.lambda_arn attribute - Add _start_mtrl_execution with proper pipeline tagging for get_all discovery (uses boto3 directly since Job step type requires beta endpoint for CreatePipeline validation) - Reuse get_presigned_mlflow_experiment_url in trainer_wait.py and multi_turn_rl_evaluator.py (DRY) - Delete unused multi_turn_rl_evaluator_utils.py (dead code after show_results removal) - Add TODO comment on custom botocore loader in utils.py * Change revert * docs: Add 3P agent (Lambda) evaluation section to notebook Add Case 4 demonstrating Lambda-based agent evaluation with: - Lambda ARN string as agent_config - AgentLambda object as agent_config - AgentLambda.create() inline code example All examples include wait() for completion. * fix: Always include MlflowExperimentName in JobConfigDocument The backend requires MlflowExperimentName when ModelPackageConfig is not provided (base model only evaluation). Default to 'mtrl-eval-{model_name}' when not explicitly set by the user. * docs: Add MTRL dogfooding notebook with Train/Eval/Deploy flows Covers three scenarios: 1. Bedrock AgentCore: train → evaluate → deploy via ModelBuilder 2. 3P Lambda agent: train → evaluate → deploy via ModelBuilder 3. Base model evaluation (no training, AgentCore + Lambda) Includes discovery utilities and cleanup section. * docs: Add Bedrock deployment section to dogfooding notebook Adds BedrockModelBuilder deploy examples for both AgentCore and Lambda training outputs, plus Bedrock runtime invocation example. * docs: Set minimal hyperparameters in dogfooding notebook Use num_epochs=1, global_batch_size=2, max_steps=5 for fast dogfooding runs instead of defaults that take hours. * fix: Use max_epochs instead of num_epochs in dogfooding notebook * chore: Disable telemetry S3 request during dogfooding The sm-pysdk-t S3 bucket is unreachable from beta accounts, causing noisy RequestException logs. Disabled until post-launch. * Revert "chore: Disable telemetry S3 request during dogfooding" This reverts commit 689c05b5fbd1aa3e8cf40b1a1b17d4f843dfcec9. * fix(serve): Fall back to hub_content_name when recipe_name is empty MTRL-trained model packages have hub_content_name set but recipe_name empty. ModelBuilder now falls back to hub_content_name for recipe lookup in the hub document, enabling deployment of MTRL fine-tuned models via ModelBuilder. * docs: Use openai-reasoning-gpt-oss-20b and add list_supported_models section Switch dogfooding notebook to openai-reasoning-gpt-oss-20b model and add a prominent Supported Models section showing both training and evaluation model discovery. * Address PR comments * Fixes --------- Co-authored-by: jamesfyu <jamesfyu@amazon.com> Co-authored-by: Bassem Halim <bassemamir459@gmail.com> Co-authored-by: Aditi Sharma <165942273+Aditi2424@users.noreply.github.com> Co-authored-by: adishaa <adishaa@amazon.com> Co-authored-by: Basssem Halim <bhhalim@amazon.com> Co-authored-by: Molly He <mollyhe@amazon.com> Co-authored-by: Zachary David Saunders <zsaund@amazon.com> Co-authored-by: Bobby Lindsey <bobbywlindsey@users.noreply.github.com> Co-authored-by: Amit <modi.osu@gmail.com> Co-authored-by: Amit Modi <modiamit@amazon.com> Co-authored-by: Rohit Kumar Srivastava <141.srivastava@gmail.com> Co-authored-by: IshaChid76 <49986634+IshaChid76@users.noreply.github.com> Co-authored-by: jam-jee <jamjee@amazon.com> Co-authored-by: rsareddy0329 <rsareddy0329@gmail.com> Co-authored-by: Joshua Towner <jjtowner@amazon.com> Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com> Co-authored-by: David Lindskog <davlind@amazon.com> Co-authored-by: mnganesh-amzn <mnganesh@amazon.com> Co-authored-by: aviruthen <91846056+aviruthen@users.noreply.github.com> Co-authored-by: Ryan <ryantanaka.y@gmail.com> Co-authored-by: Ryan Tanaka <rrtanaka@amazon.com> Co-authored-by: ampleh <22372465+ampleh@users.noreply.github.com> Co-authored-by: houtampl <houtampl@amazon.com> Co-authored-by: Syed Mujtaba <42322958+mujtaba1747@users.noreply.github.com> Co-authored-by: Mike Shen <109769013+xiaoxshe@users.noreply.github.com> Co-authored-by: Tritin Truong <tttritin@amazon.com> Co-authored-by: Barret Pickett <mrpic@amazon.com> * Update pr-checks-master.yml (#1995) * Test fixes (#1996) * feat: upgrade MultiTurnRLTrainer experience for mlflow, model package, and add integ test and SDK Docs (#1993) * feat: update MultiTurnRLTrainer and example notebook (#2002) * feat: update MultiTurnRLTrainer and example notebook * Add dataset format docs and support .json/.csv extensions for RFT - Add dataset format requirements summary to notebook section 3 - Expand DATASET_SUPPORTED_EXTENSIONS to include .json and .csv - Add _validate_csv and _validate_json to DatasetFormatDetector - Update unit tests to reflect new supported extensions * fix: subtract 1 from current step in training progress bar to avoid premature 100% (#2008) The progress bar showed 100% while the training step was still in progress because CurrentStep reports the step being worked on, not yet completed. Subtract 1 (clamped to 0) from the numerator so the bar only reaches 100% after the step actually finishes. * Mtrl readiness (#1998) * Test fixes * Test Fixes * trigger CI * fix: MTRL integ test import typo and mlflow ARN format - Fix CustomCustomAgentLambda → CustomAgentLambda import in 3p agent test - Update mlflow-tracking-server ARNs to mlflow-app format to match new validator * fix: update unit tests to match refactored MLflow boto3 API Tests in test_finetune_utils.py were referencing removed functions (_mlflow_version_meets_minimum, _wait_for_mlflow_app_ready) and old MlflowApp resource-object patterns. Updated to use the new dict-based boto3 client approach (_mlflow_version_meets_minimum_dict, _wait_for_mlflow_app_ready_boto, _get_prod_sm_client with paginator). * fix: pass explicit boto_session with region to Session() in evaluator The BaseEvaluator._create_default_session validator was creating a boto3.client with region but passing only sagemaker_client to Session(). Session.__init__ creates its own boto3.Session() without region, which fails in CI where no ~/.aws/config exists. Fix by creating a boto3.Session(region_name=region) first and passing it as boto_session. Also pass explicit sagemaker_session to MultiTurnRLTrainer in the test_mtrl_evaluator.py fixture to avoid the same issue in the trainer constructor. * revert: remove version bumps from this branch Reverts VERSION files back to 2.10.0 (core) and 1.10.0 (train, serve), and removes the data/sample package-data glob from pyproject.toml. * fix: use dynamic account ID in integ tests instead of hardcoded 742774200982 Resolves current account via STS at module load time so tests work in any CI account without cross-account S3 permission errors. * fix: make MTRL evaluator integ tests fully account-agnostic - Model resolution now uses pre-resolved _model_arn/_model_name from trainer when available, avoiding DescribeModelPackage calls for trainers that already have this info cached. - Test fixture creates model package group on the fly instead of relying on pre-existing resources in a specific account. - All hardcoded account IDs replaced with dynamic STS resolution. * fix: remove end-to-end wait tests that require account-specific resources These tests called execution.wait() and asserted Succeeded, which requires real Bedrock AgentCore runtimes and trained model artifacts. The pipeline submission flow is already covered by test_evaluate_comparison_mode and test_pipeline_reuse. * fix: replace pipeline submission tests with construction tests Account 391266019386 does not support the Job step type in SageMaker Pipelines, so pipeline creation/execution tests cannot run. Replaced with evaluator construction tests that validate the SDK code path (model resolution, session creation, validator logic) without submitting pipelines. * test: add unit tests for model builder is_checkpoint and IC model_name changes Cover the new is_checkpoint logic in _resolve_model_artifact_uri, _fetch_peft, build(), and inference component creation using model_name. * feat: create restricted model package group for Nova models (#2013) * chore: update svc model (#2005) * feat: create restricted model package group for Nova models When MultiTurnRLTrainer auto-creates a model package group for Nova models, pass ManagedConfiguration(managed_storage_type="Restricted") to create a restricted MPG. Applies to both output and intermediate checkpoint MPGs. --------- Co-authored-by: Syed Mujtaba <42322958+mujtaba1747@users.noreply.github.com> * Mtrl readiness (#2019) * Test fixes * Test Fixes * trigger CI * fix: MTRL integ test import typo and mlflow ARN format - Fix CustomCustomAgentLambda → CustomAgentLambda import in 3p agent test - Update mlflow-tracking-server ARNs to mlflow-app format to match new validator * fix: update unit tests to match refactored MLflow boto3 API Tests in test_finetune_utils.py were referencing removed functions (_mlflow_version_meets_minimum, _wait_for_mlflow_app_ready) and old MlflowApp resource-object patterns. Updated to use the new dict-based boto3 client approach (_mlflow_version_meets_minimum_dict, _wait_for_mlflow_app_ready_boto, _get_prod_sm_client with paginator). * fix: pass explicit boto_session with region to Session() in evaluator The BaseEvaluator._create_default_session validator was creating a boto3.client with region but passing only sagemaker_client to Session(). Session.__init__ creates its own boto3.Session() without region, which fails in CI where no ~/.aws/config exists. Fix by creating a boto3.Session(region_name=region) first and passing it as boto_session. Also pass explicit sagemaker_session to MultiTurnRLTrainer in the test_mtrl_evaluator.py fixture to avoid the same issue in the trainer constructor. * revert: remove version bumps from this branch Reverts VERSION files back to 2.10.0 (core) and 1.10.0 (train, serve), and removes the data/sample package-data glob from pyproject.toml. * fix: use dynamic account ID in integ tests instead of hardcoded 742774200982 Resolves current account via STS at module load time so tests work in any CI account without cross-account S3 permission errors. * fix: make MTRL evaluator integ tests fully account-agnostic - Model resolution now uses pre-resolved _model_arn/_model_name from trainer when available, avoiding DescribeModelPackage calls for trainers that already have this info cached. - Test fixture creates model package group on the fly instead of relying on pre-existing resources in a specific account. - All hardcoded account IDs replaced with dynamic STS resolution. * fix: remove end-to-end wait tests that require account-specific resources These tests called execution.wait() and asserted Succeeded, which requires real Bedrock AgentCore runtimes and trained model artifacts. The pipeline submission flow is already covered by test_evaluate_comparison_mode and test_pipeline_reuse. * fix: replace pipeline submission tests with construction tests Account 391266019386 does not support the Job step type in SageMaker Pipelines, so pipeline creation/execution tests cannot run. Replaced with evaluator construction tests that validate the SDK code path (model resolution, session creation, validator logic) without submitting pipelines. * test: add unit tests for model builder is_checkpoint and IC model_name changes Cover the new is_checkpoint logic in _resolve_model_artifact_uri, _fetch_peft, build(), and inference component creation using model_name. * fix: merged model deployment and MLflow deep-linking for eval ModelBuilder (SMI): - Add is_checkpoint check in _fetch_peft() to skip LORA path for merged models - Resolve merged model artifacts to checkpoints/hf_merged/ in build and deploy - Use model_name instead of artifact_url in non-LORA IC spec MLflow URL deep-linking: - Use deepLink query param (matching SageMaker UI pattern) instead of URL fragments - Authenticate via presigned URL session to resolve experiment name to ID - Store mlflow_resource_arn and mlflow_experiment_name on eval execution - Fall back to experiment name search filter when ID resolution fails BedrockModelBuilder: - Add fallback to fetch output_model_package_arn from job config when not set * Resolving unit test failures * Fix test_merged_model_deployment isinstance compatibility across Python versions --------- Co-authored-by: Ming Luo <24469267+mingluo0108@users.noreply.github.com> Co-authored-by: jamesfyu <jamesfyu@amazon.com> Co-authored-by: Bassem Halim <bassemamir459@gmail.com> Co-authored-by: Aditi Sharma <165942273+Aditi2424@users.noreply.github.com> Co-authored-by: adishaa <adishaa@amazon.com> Co-authored-by: Basssem Halim <bhhalim@amazon.com> Co-authored-by: Molly He <mollyhe@amazon.com> Co-authored-by: Zachary David Saunders <zsaund@amazon.com> Co-authored-by: Bobby Lindsey <bobbywlindsey@users.noreply.github.com> Co-authored-by: Amit <modi.osu@gmail.com> Co-authored-by: Amit Modi <modiamit@amazon.com> Co-authored-by: Rohit Kumar Srivastava <141.srivastava@gmail.com> Co-authored-by: IshaChid76 <49986634+IshaChid76@users.noreply.github.com> Co-authored-by: jam-jee <jamjee@amazon.com> Co-authored-by: rsareddy0329 <rsareddy0329@gmail.com> Co-authored-by: Joshua Towner <jjtowner@amazon.com> Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com> Co-authored-by: David Lindskog <davlind@amazon.com> Co-authored-by: mnganesh-amzn <mnganesh@amazon.com> Co-authored-by: aviruthen <91846056+aviruthen@users.noreply.github.com> Co-authored-by: Ryan <ryantanaka.y@gmail.com> Co-authored-by: Ryan Tanaka <rrtanaka@amazon.com> Co-authored-by: ampleh <22372465+ampleh@users.noreply.github.com> Co-authored-by: houtampl <houtampl@amazon.com> Co-authored-by: Syed Mujtaba <42322958+mujtaba1747@users.noreply.github.com> Co-authored-by: Mike Shen <109769013+xiaoxshe@users.noreply.github.com> Co-authored-by: Tritin Truong <tttritin@amazon.com> Co-authored-by: Barret Pickett <mrpic@amazon.com> * feat: Support RMP for serverful and serverless Training Job (#2010) - Add model_package_config field to ModelTrainer for RMP consumption - Route MP ARN from recipe to ModelPackageConfig.SourceModelPackageArn - Add S3Uri optional override for escrow-managed artifacts (RMP) - Fix duplicate min_memory_required_in_mb in codegen output - Add studio_web_portal_settings to codegen output (missed in #2018) - Direct model_package_config param overrides recipe (CTJ priority) Reverted: ModelPackageGroupArn remains required (pending Smithy approval) Test results: - sagemaker-train unit tests: 81/81 passed - sagemaker-core unit tests: 3217/3225 passed - sagemaker-core tools tests: 39/39 passed - 8 expected failures (Docker Compose not installed locally, no code changed in tests/unit/local/): tests/unit/local/test_image.py::TestSageMakerContainerAdvanced (all 8) Co-authored-by: xibei chen <xibeich@amazon.com> * Add inference component discovery and bedrock deploy polling (#2020) * Add inference component discovery and bedrock deploy polling - Add cell to list inference components for an endpoint before invoke - Add polling cells to wait for bedrock deployment to reach InService status before invoking (both Scenario 1 and Scenario 2) * Replace internal dogfooding notebook with open-source MTRL example - Remove account-specific ARNs, gamma endpoints, and internal references - Consolidate duplicate scenarios into a single clean flow (train → eval → deploy) - Add both SageMaker endpoint and Bedrock deployment paths - Include inference component discovery and deploy polling - Add descriptive markdown cells explaining each step - Move to model-customization-examples directory * Add evaluation and deployment sections to MTRL prod notebook - Add Section 11: Evaluate fine-tuned model and base model comparison - Add Section 12: Deploy to SageMaker endpoint with inference component discovery and invocation - Add Section 13: Deploy to Bedrock with polling and invocation - Add Section 14: Cleanup - Remove account-specific info from setup cell (ada credentials, internal paths) - Remove standalone mtrl_finetuning_example_notebook (consolidated here) - Update table of contents to reflect new sections * feat: add run-level MLflow deep-linking for MTRL eval execution (#2028) * feat: add run-level MLflow deep-linking for MTRL eval execution The eval MLflow URL now deep-links to the specific run (experiment_id + run_id), matching the training job behavior. Previously it only linked to the experiment level. Changes: - Add get_mlflow_url() and get_mlflow_details() to MTRLEvaluationExecution - Add get_presigned_mlflow_url() and _resolve_run_id() to mlflow_url_utils - Fix URL format to use fragment-based routing (#/experiments/id/runs/id) with ?workspace=default for SageMaker MLflow app compatibility - Refresh presigned URL every 30s during wait() (matching trainer behavior) - Add demo notebook for the feature * revert: remove demo notebooks from PR The notebooks are for local testing only, not needed in the PR. * Update MTRL trainer integ test (#2029) * Port Integ tests to SDK infra account * Port Integ tests to SDK infra account --------- Co-authored-by: Gokul Anantha Narayanan <166456257+nargokul@users.noreply.github.com> Co-authored-by: Syed Mujtaba <42322958+mujtaba1747@users.noreply.github.com> Co-authored-by: Jonathan Makunga <54963715+makungaj1@users.noreply.github.com> Co-authored-by: Jonathan makunga <makung@amazon.com> Co-authored-by: ABICHAL GHOSH <42579402+ABICHAL1708@users.noreply.github.com> Co-authored-by: Mike Shen <109769013+xiaoxshe@users.noreply.github.com> Co-authored-by: Tritin Truong <tttritin@amazon.com> Co-authored-by: Barret Pickett <mrpic@amazon.com> Co-authored-by: James Yu <jamesfyu@amazon.com> Co-authored-by: Mahima Chaudhary <36598677+cmahima@users.noreply.github.com> Co-authored-by: Mahima Chaudhary <mahchy@amazon.com> Co-authored-by: Ming Luo <24469267+mingluo0108@users.noreply.github.com> Co-authored-by: Bassem Halim <bassemamir459@gmail.com> Co-authored-by: Aditi Sharma <165942273+Aditi2424@users.noreply.github.com> Co-authored-by: adishaa <adishaa@amazon.com> Co-authored-by: Basssem Halim <bhhalim@amazon.com> Co-authored-by: Molly He <mollyhe@amazon.com> Co-authored-by: Zachary David Saunders <zsaund@amazon.com> Co-authored-by: Bobby Lindsey <bobbywlindsey@users.noreply.github.com> Co-authored-by: Amit <modi.osu@gmail.com> Co-authored-by: Amit Modi <modiamit@amazon.com> Co-authored-by: Rohit Kumar Srivastava <141.srivastava@gmail.com> Co-authored-by: IshaChid76 <49986634+IshaChid76@users.noreply.github.com> Co-authored-by: jam-jee <jamjee@amazon.com> Co-authored-by: Joshua Towner <jjtowner@amazon.com> Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com> Co-authored-by: David Lindskog <davlind@amazon.com> Co-authored-by: mnganesh-amzn <mnganesh@amazon.com> Co-authored-by: aviruthen <91846056+aviruthen@users.noreply.github.com> Co-authored-by: Ryan <ryantanaka.y@gmail.com> Co-authored-by: Ryan Tanaka <rrtanaka@amazon.com> Co-authored-by: ampleh <22372465+ampleh@users.noreply.github.com> Co-authored-by: houtampl <houtampl@amazon.com> Co-authored-by: sylvie7788 <43765909+sylvie7788@users.noreply.github.com> Co-authored-by: xibei chen <xibeich@amazon.com>rsareddy0329 · 213cc61b · 2026-06-03
- 3.9ETVFeature store v3 (#5539) * Feature store v3 (#5490) * feat: Add Feature Store Support to V3 * Add feature store tests --------- Co-authored-by: adishaa <adishaa@amazon.com> * Changes: - Add feature_store_functions_report.md documenting all 63 functions across feature_store module (excluding feature_processor) - Add comprehensive unit tests for get_feature_group_as_dataframe: * Session handling (provided, from region, from role) * Error cases (missing session/region, missing event_time) * Latest ingestion logic with event time * Query string manipulation and table placeholder * Verbose and silent logging modes * Kwargs passing to as_dataframe - Add comprehensive unit tests for prepare_fg_from_dataframe_or_file: * DataFrame and file path input handling * Session/region/role configuration * Record ID creation and validation * Event ID creation with timestamp * Duplicate record detection * Column name formatting * CSV kwargs passing * Feature definition loading * Fix * Fix * fix * Integ tests * Fix * Add feature store telemetry * Bug fix for GetRecordResponse not printable and error message for ingest_dataframe * Fix: DatasetBuilder.to_dataframe() docstring and instantiation * Fix unit test failures * fix unit test --------- Co-authored-by: Aditi Sharma <165942273+Aditi2424@users.noreply.github.com> Co-authored-by: adishaa <adishaa@amazon.com> Co-authored-by: Molly He <mollyhe@amazon.com>Gokul Anantha Narayanan · 487764ae · 2026-02-27
- 3.1ETVBug Fixes Model Customization (#5558) * Bug Fixes Model Customization * fixes * Fix * Fix * Fix * Test FIxesGokul Anantha Narayanan · c72a77d8 · 2026-02-20
- 2.1ETVchange(train): gate deep integ tests behind gpu_intensive, add shallow submit-then-stop suite (#6176) * change(train): gate deep integ tests behind gpu_intensive, add shallow submit-then-stop suite Replaces the CodeBuild integ suite for sagemaker-train on the PR gate with a faster selection that keeps meaningful server-side coverage. CreateTrainingJob returns a TrainingJobArn only after the request has cleared every synchronous server-side gate: public-model shape validation, SigV4, sagemaker:CreateTrainingJob authorization (including condition keys), iam:PassRole on the execution role, the training backend's synchronous request validators, its role-assuming validators (which make real S3/ECR/FSx calls as the customer), post-validator business logic (training-plan capacity, routing, recipe filtering) and the final conditional write that rejects duplicate job names. So "the ARN came back" proves the SDK-shaped payload was accepted as sent and the caller held the permissions needed to submit it -- without paying for a training run. Adds tests/integ/train/shallow (70 tests) built on that: submit, assert the ARN, stop immediately. Covers ModelTrainer (payload shaping, source-code packaging, input channels, compute, networking, checkpointing/spot), the recipe trainers (SFT/DPO/RLVR/RLAIF, serverless and serverful), recipe customization (overrides, explicit recipe files, sequence_length, DataMixingConfig), and the non-training job types (HyperParameterTuningJob, AgentRFT Job). Includes negative tests so the suite cannot pass merely because some ARN came back. Marks the 19 previously-unmarked tests that submit a job and wait for it with gpu_intensive, so they continue running on the scheduled CI-health workflows instead of the PR gate. Widens that marker's description: despite the name it gates anything consuming real training capacity, including serverless and CPU-instance jobs. The PR job now runs the whole tests/integ/train tree with -m "not gpu_intensive and not us_east_1" rather than only shallow/, which keeps the ~170 client-side tests (recipe resolution, data utils, dry-run, log streaming) on the gate -- they make no service call and were never the expensive part. Net: 191 of 251 tests on the PR gate, none of which waits for a training job. This is a deliberate scope reduction: training *behaviour* (artifacts, metrics, convergence) is no longer asserted on the PR gate. A regression that breaks training itself -- a bad entry script, a broken container command -- will pass here and be caught by the scheduled suites. * change(train): fix CPTTrainer construction and role-rejection test after first real AWS run Verified against AWS in account 729646638167 (us-west-2): * test_unassumable_role_is_rejected: ModelTrainer.__init__ validates the role via iam:SimulatePrincipalPolicy, so a bad role raises RoleValidationError at construction and never reaches CreateTrainingJob. Assert around the constructor instead of around train(). * test_cpt_trainer_is_accepted: CPTTrainer takes no training_type, and its compute is HyperPodCompute-only, so it cannot use the shared _trainer helper. WIP: 2 further real failures still to fix (RLAIF compute, tuner job-name collision). See SHALLOW_TEST_RUN_STATE.md. * change(train): fix shallow suite against real AWS; 62/62 passing Ran the suite against account 729646638167 (us-west-2) with PYTHONPATH pointed at this clone, and fixed every failure it surfaced. All were wrong assumptions in the tests, not service problems: * conftest: add a session-scoped bundled_service_model fixture setting AWS_DATA_PATH to sagemaker-core/sample. The public botocore model has no ServerlessJobConfig.SequenceLength, so sequence_length requests were rejected client-side before reaching the service. Mirrors the existing setup_aws_data_path fixture in test_recipe_override_integration.py. * harness: unique_name() now takes max_length. Tuning job names are capped at 32 characters, not the 63 allowed for training jobs, and the service enforces it: Value '...' at 'hyperParameterTuningJobName' failed to satisfy constraint: Member must have length less than or equal to 32 * tuner tests: submit under an explicit job_name via a _tuning() context manager. The tuner derives its default name from the training image plus a second-granularity timestamp and ignores base_job_name, so two tuner tests in the same second collided with ResourceInUse. * RLAIF: excluded from TestServerfulSubmission. RLAIFTrainer has no compute parameter, so it has no serverful path. Still covered by every serverless case. * CPT: marked gpu_intensive and skipped unless SHALLOW_HYPERPOD_CLUSTER is set. CPT refuses to submit without HyperPod compute, and HyperPod targets a pre-provisioned cluster rather than CreateTrainingJob. * sequence_length / training_type: narrowed to the values the recipe catalogue actually offers for this model ('4K' only; no serverless recipe for FULL). Both left parametrized so more values can be added against a model that supports them, rather than dropping the distinction. Result: 62 passed, 0 failed, 5m18s serial (~5s/test). Cost model confirmed empirically rather than assumed: across 100 jobs created by these runs, every one ended Stopped and every BillableTimeInSeconds was null. Jobs are torn down while still in Starting/Pending, before instances become billable. * change(train): one shallow file per trainer; only mark deep tests that have shallow coverage Addresses two review points. 1. Only mark deep tests that this suite actually replaces. Reverts gpu_intensive from 9 tests that had no shallow counterpart, so the PR gate no longer loses coverage with nothing replacing it: * all 8 evaluator tests (benchmark, custom scorer, inspect_ai, llm_as_judge x2, llmaj_custom_model) -- evaluate() is a different API surface returning pipeline executions, and this suite has no coverage for it * test_notifications.py -- asserts EventBridge/SNS side effects, not submission 10 marks remain, each with a named shallow equivalent documented in the suite README. The rule is written down there: do not mark a deep test unless a shallow test covers the same path. 2. One file per trainer, matching the existing deep-suite layout. test_recipe_trainers_submission.py -> test_{sft,dpo,rlvr,rlaif,cpt}_trainer.py test_recipe_customization_submission.py (recipe cases folded into rlvr/sft; Nova data mixing to its own file) test_other_job_types_submission.py -> test_tuner.py, test_multi_turn_rl_trainer.py test_model_trainer_submission.py -> test_model_trainer.py The "recipe_*" names described how the SDK groups these internally rather than what a reader looks for; the shallow counterpart of a given deep test is now obvious from the filename. recipe_cases.py holds the cases every recipe trainer shares. Each per-trainer class subclasses RecipeTrainerCases and sets TRAINER, so a new trainer is a two-line file, and per-trainer deviations are declared rather than duplicated: EXTRA_KWARGS (RLAIF's reward model), SUPPORTS_SERVERFUL=False (RLAIF takes no compute), SUPPORTS_TRAINING_TYPE=False (CPT has no LoRA/full split). Not named test_* so pytest does not collect the base class. Inheriting the shared cases also widened coverage: DPO and RLAIF now get the full set (output path, dataset override, both negative cases) rather than only the three they had as parametrized entries. 80 tests total, 69 on the PR gate. Verified against AWS (account 729646638167, us-west-2): 68 passed, 1 skipped, 0 failed in 6m59s. The skip is RLAIF's serverful case, reporting "RLAIFTrainer takes no compute argument". * change(train): add shallow coverage for every gpu_intensive test that has an equivalent Previous commits only audited the marks this PR added. This audits all 46 gpu_intensive tests in tests/integ/train -- including those already marked on master -- and adds the missing shallow counterparts. Added (were gaps): * MLflow, in RecipeTrainerCases so all four recipe trainers get it. Every *_complete_workflow deep test configures MLflow, so without this their shallow counterparts missed that half of the payload. Two forms: experiment/run names (always runs) and mlflow_resource_arn (skips if the account has no app). * RLVR reward functions, all three forms the deep suite covers: hub-content ARN, Lambda ARN (auto-creates an Evaluator), and a pre-created Evaluator object. * RLAIF reward_prompt as a hub-content ARN rather than a Builtin.* name, and continued fine-tuning from a model-package ARN. * Nova SFT and Nova RLVR, in test_nova_trainers.py. Nova needs a different recipe family, region and account, so it cannot share RecipeTrainerCases; marked us_east_1. Two real constraints the AWS run surfaced, both now recorded in comments: * The reward-function tests cannot use this suite's generic chat-format fixture. Before submitting, the SDK *invokes* the reward function over sample records and fails if they do not score ("GSM8k scoring failed"). They now use the same dataset as the deep RLVR suite, via a dedicated reward_scored_data_uri fixture. * list_mlflow_apps is not a paginatable operation, so the fixture calls it directly instead of via get_paginator. Also fixed a ScopeMismatch: the three new lookup fixtures were session-scoped but depend on the parent conftest's module-scoped sagemaker_session. All three new fixtures (mlflow_arn, reward_lambda_arn, reward_evaluator) only look resources up and skip when absent. The deep suite's equivalents create them -- IAM roles, Lambdas, MLflow apps, registry entries -- which is a durable side effect a fast PR-gate suite should not have. Still uncovered, documented in the suite README with the reason: the 11 evaluator tests (evaluate() is a different API surface returning pipeline executions) and the 3 HyperPod tests (submit to a pre-provisioned cluster, not CreateTrainingJob). Neither is newly marked by this PR, so no coverage is lost; the evaluator gap is the clearest follow-up. 97 tests total, 82 on the PR gate. Verified against AWS (729646638167, us-west-2): 81 passed, 1 skipped, 0 failed in 7m04s. The skip is RLAIF's serverful case, which reports its own reason. * change(train): add shallow coverage for recipe overrides, GRPO hyperparameters, Nova serverful Three remaining gpu_intensive tests had no shallow counterpart: * test_sft_trainer_serverful_smtj.py (override half) -> SFT test_recipe_overrides_are_accepted. Asserts both halves: the merge reached the rendered recipe (client-side, exact) and the resulting payload is still accepted (recipe filtering runs after the request validators, so a bad merge only surfaces at submission). Verified against AWS: overrides are written flat under training_config but land nested under training_args, and the recipe default for this model is 5 -- so asserting 1 proves the override applied rather than coinciding with the default. * test_rlvr_trainer_nemotron_with_kl_and_recipe -> RLVR test_kl_and_clipping_hyperparameters. These are separate recipe fields rather than one flag, so the existing max_epochs-only test did not prove they serialize. * test_sft_trainer_serverful_smtj.py (Nova half) -> Nova TestNovaServerfulSubmission. Distinct from the shared serverful case: Nova model, Nova recipe family, Nova-only instance type, us-east-1. Accepts the override under either trainer.max_epochs or training_args.max_epochs, since recipe families nest epoch control differently -- so the test fails on a lost override rather than on a recipe-layout difference. Verified against a real account (us-west-2): 83 passed, 1 skipped, 0 failed in 5m20s. The skip reports its own reason (RLAIFTrainer takes no compute argument). * fix(train): make the shallow Nova tests runnable in any account The five us_east_1 shallow tests referenced resources hardcoded to one test account and had therefore never actually executed. Verified: from 729646638167, `aws s3 ls s3://sagemaker-us-east-1-784379639078/input_data/sft-nova/` returns AccessDenied. Derive everything from the calling account instead, the way test_sft_trainer_serverful_smtj.py::training_resources already does: * nova_sft_data_uri -- uploads the Nova-shaped sample data the deep suite already ships (tests/data/train/sft_smtj_sample_data.jsonl) to the caller's own bucket. Cannot reuse nova_train_data_uri: Nova SFT records carry a schemaVersion the generic chat-format fixture lacks. * nova_rlvr_data_uri -- copies the GSM8k-shaped dataset the us-west-2 RLVR tests use into the us-east-1 bucket. A copy rather than a reference because an S3 input must be in the job's region. * nova_output_path -- default_bucket() rather than a named bucket. * nova_reward_function_arn -- resolves the hub content in the caller's own account, look-up-and-skip like the other reward fixtures. Two service-verified region constraints drove this: * the model package group must be in the job's region -- passing the us-west-2 MODEL_PACKAGE_GROUP ARN is rejected with "Model package group ARN region 'us-west-2' does not match expected region 'us-east-1'". Added NOVA_MODEL_PACKAGE_GROUP (a bare name) alongside it in recipe_cases so the two Nova files cannot drift. * likewise for S3 inputs, hence the RLVR copy above. The Nova RLVR case sets skip_reward_validation=True. The SDK invokes the reward function over sample records before submitting; the function registered under that name in this account returns a shape the verifier rejects ("Each output must include 'id', 'aggregate_reward_score'"), so the test would assert per-account hub contents rather than this payload. The verifier is already covered against a known-compatible function by the three us-west-2 reward-function cases; what is unique here is the Nova recipe family and region. Also register gpu_intensive and us_east_1 in pyproject.toml. They were declared only in tox.ini, but pytest reads its config from pyproject.toml, so both were unregistered at runtime. That matters here: the PR gate selects with -m "not gpu_intensive and not us_east_1", so a typo'd marker name would silently put an expensive deep test back on the gate instead of warning. Verified against a real account: 5 passed in 47s, all five for the first time. Every job ended Stopped with BillableTimeInSeconds null, so the cost model holds in us-east-1 as well. * docs(train): record what actually bounds the PR gate's runtime A full gate run showed the shallow suite is not what makes this job slow. Measured (us-west-2, -n 8 --dist loadfile): 201 of 204 tests finished in ~7 minutes, then three evaluator tests held the run open for another 40+ before being killed. Five evaluator tests are not marked gpu_intensive and each blocks on execution.wait(..., timeout=14400) -- a 4-hour ceiling, ~33 minutes per execution in practice: test_benchmark_evaluator.py::test_benchmark_evaluation_full_flow (no marks) test_custom_scorer_evaluator.py::test_custom_scorer_evaluation_full_flow (xdist_group) test_llm_as_judge_evaluator.py::test_llm_as_judge_evaluation_full_flow (no marks) test_llm_as_judge_base_model_fix.py::test_base_model_evaluation_uses_correct_weights (serial) test_llm_as_judge_base_model_fix.py::test_base_model_false_still_works (serial) They run on master's gate too, so this PR does not add them -- but it does not fix them either, and they now dominate the job's wall clock. Deliberately NOT marking them here: unlike every other gpu_intensive test they have no shallow counterpart, so marking would remove coverage, which is what the rule this PR establishes forbids. Correct order is to add evaluator support to the harness first, then mark. Documented in the suite README so the next person does not have to rediscover it by watching a run stall at 95%. Also flags test_local_model_trainer.py in the workflow: it runs real containers, so it needs Docker and pulls pytorch-training:2.0.0-cpu-py310 (2.3 GB compressed, verified via ECR). That is fine on GitHub-hosted Ubuntu runners, which preinstall Docker, and the ECR read is already covered by the role the shallow tests use -- but it is the slowest non-evaluator thing on the gate and the only step with a disk-space floor, so the note says what to deselect first if the job ever goes flaky on runner capacity. * change(ci): keep the sagemaker-train integ job, add shallow tests alongside it Restores integ-tests to its master definition -- sagemaker-train is back in the matrix, byte-identical to master -- and makes fast-integ-tests additive rather than a replacement. The deep tests still come off the gate, just not by removing the job. The CodeBuild project's buildspec already selects -m "not gpu_intensive and not us_east_1" (verified by reading the live project), so the marks added earlier in this PR are what deselect them. No workflow edit was needed for that. Keeping the CodeBuild job also keeps things the shallow job cannot cover: * the whole tests/integ tree, so the ~170 client-side tests (recipe resolution, data utils, dry-run, log streaming) run without this job repeating them; * test_local_model_trainer.py, which needs a Docker daemon. CodeBuild runs start-dockerd with privilegedMode, which is a better home for it than a GitHub runner pulling a 2.3 GB image -- so the reviewer caveat about that is dropped as moot; * the serial/parallel split the buildspec does for rate-limited tests. fast-integ-tests is therefore scoped to tests/integ/train/shallow only. Widening it would duplicate the client-side tests and double the training jobs this suite creates. It stays a separate job rather than folding into the buildspec because the buildspec is CDK-managed outside this repo, and because a shallow failure then reports as its own check. Corrects a claim in the previous comment: the shallow suite does carry gpu_intensive tests -- 11 of them, the CPT and MTRL classes, which need a HyperPod cluster and an agent runtime. With us_east_1 that is 16 deselected, so 84 of 100 run here. The comment now lists both groups and why. Verified against a real account: 83 passed, 1 skipped, 0 failed in 3m15s (the skip self-reports: RLAIFTrainer takes no compute argument). Faster than the 5m20s measured with the client-side tests bundled in. Every job ended Stopped with BillableTimeInSeconds null; no leaked jobs. * fix: memoize role validation and mark six pipeline-waiting evaluator tests Two problems the PR gate surfaced on its own run of this branch. 1. SimulatePrincipalPolicy throttling (4 shallow tests failed) FAILED tests/integ/train/shallow/test_model_trainer.py::TestSourceCodePackaging::test_shell_entry_script FAILED tests/integ/train/shallow/test_rlaif_trainer.py::TestRLAIFTrainerSubmission::test_mlflow_resource_arn FAILED tests/integ/train/shallow/test_rlvr_trainer.py::TestRLVRTrainerSubmission::test_with_validation_dataset FAILED tests/integ/train/shallow/test_rlvr_trainer.py::TestRLVRTrainerSubmission::test_dataset_passed_to_train_overrides_constructor botocore.exceptions.ClientError: An error occurred (Throttling) when calling the SimulatePrincipalPolicy operation (reached max retries: 9): Rate exceeded Not a test defect: all four pass locally in isolation and in a local -n 36 run. Every ModelTrainer construction calls TrainDefaults.get_role -> resolve_and_validate_role, which paginates SimulatePrincipalPolicy over ~20 action names against a low, account-wide TPS limit. The CodeBuild job runs the whole tests/integ tree under -n auto (~36 workers on a 2XLARGE), which is 188 trainer constructions -- enough to exhaust even the adaptive 10-attempt budget the existing _configure_boto_adaptive_retries fixture grants. The cause is volume, not burstiness, so more retries would not have fixed it. Fixed with a _memoize_role_validation autouse session fixture: each distinct (role, role_type, region) is validated once per xdist worker instead of once per test. Measured with an instrumented botocore _make_api_call: 3 trainers -> 3 calls unpatched, 10 trainers -> 1 call memoized. Two details worth keeping: * exceptions are cached alongside successes, so a bad role still fails -- test_unassumable_role_is_rejected still passes; * teardown restores any caller now holding the memoized function, not just the ones this fixture explicitly patched. A module imported after the source module was patched binds the memoized function at its own import time, so restoring only what was patched here would leak across the session. Verified: 83 passed, 1 skipped, 0 failed in 3m33s with memoization active. 2. Six evaluator tests wait on a full evaluation pipeline From the same build's serial pass durations: 2783.83s test_llm_as_judge_base_model_fix.py::test_base_model_evaluation_uses_correct_weights 2504.59s test_llm_as_judge_base_model_fix.py::test_base_model_false_still_works 91.44s the next-slowest test in that pass 88 minutes for two tests, against a 180-minute build timeout, and they were the entire tail. Each blocks on execution.wait(..., timeout=14400) -- a 4-hour ceiling per test. Marked gpu_intensive, along with the three *_full_flow tests that wait the same way in test_benchmark_evaluator.py, test_custom_scorer_evaluator.py and test_llm_as_judge_evaluator.py. test_llmaj_custom_model.py was a genuine mismarking: it carried @pytest.mark.slow, but the registered marker name is slow_test, so the mark silently did nothing (PytestUnknownMarkWarning). us_east_1 already kept it off the us-west-2 gate, so this changes nothing there; it now also stays off the us-east-1 job. This is a small, real coverage reduction, and the README says so rather than claiming otherwise. Three of the files are marked per-test and keep their constructor/validation tests on the gate; the two class-level ones leave nothing behind, and what the gate stops checking is that a submitted pipeline is accepted and succeeds. Already-marked siblings in the same files (test_benchmark_evaluation_base_model_only, test_custom_scorer_base_model_only) show this was already the established call for pipeline-waiting tests -- these six were unmarked by omission. Shallow evaluate() coverage is the follow-up that closes the gap. Verified: 266/342 collected on the gate's selection (76 deselected), none of the six selected, and no PytestUnknownMark warnings remain. * change(train): cap concurrent training jobs in the shallow suite The shallow suite creates a training job per test. That puts it against two different quotas in two different units: * serverless (the default recipe-trainer path, no explicit compute) is bounded by "Maximum number of concurrent model customization serverless jobs per Region" -- a count of jobs, currently 20; * serverful (an explicit Compute/TrainingJobCompute: the ModelTrainer tests, the tuner, test_explicit_compute_is_accepted) is bounded by the per-instance-type quota, e.g. "ml.m5.large for training job usage" -- a count of instances. A slot is one concurrent job; a serverful job also takes one per instance, so a single cap holds the suite inside both quotas without the harness needing to know which kind of job a given test produces. Hold the slot until the job is terminal, not until stop() returns This is the subtle part, and the first cut got it wrong. The service counts a job against the concurrency quota from CreateTrainingJob until the job reaches Completed/Failed/Stopped -- NOT until StopTrainingJob returns. Measured against the service, stop() returns in a few seconds but the job takes ~1-3 min to actually drain (the reservation is torn down without ever becoming billable). Releasing the slot at stop() therefore bounded nothing: with the cap at 10 and 8 workers, each slot recycled ~20x inside a single job's counted lifetime, the suite peaked at ~37 concurrent jobs, and it tripped ResourceLimitExceeded at a utilization of 21 against the limit of 20. _wait_until_terminal closes that gap by holding the slot across the drain, so the cap bounds what the service actually counts. With the fix, live counted concurrency stayed at 4-5 against a cap of 10 for the whole run. The cost is runtime: holding to terminal makes the suite's floor roughly (#jobs * drain) / cap. At ~83 jobs, a ~75s median drain and cap 10 that is ~8-13 min, versus ~2 min if slots released early -- but that fast run is the one that breaches the quota. This is the batches-of-10 behaviour: at most 10 jobs counted at once. Mechanism job_slots() in harness.py, held by submitted() and assert_rejected() until the job is terminal. Slots are O_EXCL-created files under a run-keyed temp directory; xdist workers are separate processes, so an in-process semaphore would bound nothing. Keyed on PYTEST_XDIST_TESTRUNUID (falling back to the parent pid) so two concurrent local runs get separate budgets rather than deadlocking, and a stale directory from a killed run is never mistaken for live slots. Details that matter: * both waits proceed with a warning rather than failing -- acquiring a slot waits up to 900s, _wait_until_terminal up to 300s -- since the cap is a courtesy to the quota, not an assertion about the SDK, and a leaked slot or stuck drain should mean a slower run rather than a red build; * status is read per job type (training_job_status / job_status / hyper_parameter_tuning_job_status), since the SDK is not consistent, and a job that exposes no status releases its slot immediately rather than hanging; * a request larger than the cap is clamped, so a single test cannot deadlock against itself; * enforced in the harness rather than per test, so a new test is capped by default instead of by remembering to opt in. Default 10, overridable via SHALLOW_MAX_CONCURRENT_JOBS; 0 disables gating for a single-worker debugging run. Set explicitly in the workflow so the ceiling is visible at the call site rather than only in a Python default. Verified * Slot mechanism holds under contention: 12 processes x 4 iterations against cap=3, observed peak exactly 3, never 4; slots released on the happy path, on exception, and with correct multi-slot accounting; cap=0 takes none; an oversized request clamps without deadlocking. * Terminal-hold bounds what the service counts: a multi-process simulation where each job stays "counted" past stop() peaked at exactly the cap (3) with 10 workers, versus the pre-fix design that would have peaked far higher. * _wait_until_terminal waits through non-terminal states, releases on terminal, honours each job type's status attribute, and returns rather than hanging on None / a read error / a timeout. * Full suite green with the fix: 83 passed, 1 skipped in 810s (13:30), zero ResourceLimitExceeded, live counted concurrency 4-5 throughout, account fully drained afterward. * docs(train): drop account IDs from the shallow suite's comments This is a public repo, so the comments should not name internal test accounts. Every reference was explanatory -- "the deep test hardcodes a bucket in account X, which other accounts cannot read" -- and the point it makes is that the bucket belongs to *one specific account*, not which account that is. Reworded to say that instead, keeping each rationale (and the verified AccessDenied finding) intact. Comments and docs only; no functional change. Test resource ARNs still name the account they actually live in, since resolving them is what the tests do, and that already matches the convention in the surrounding suite. * change(train): use a bare model package group name in both regions Review feedback: recipe_cases.py pinned MODEL_PACKAGE_GROUP to a full ARN while the Nova path used a bare NOVA_MODEL_PACKAGE_GROUP, for the same group. The bare name is the better form on both paths, so the two constants collapse into one. The SDK accepts either -- _resolve_model_package_group_arn() returns an ARN unchanged and otherwise resolves a name via ModelPackageGroup.get() against the *session's* region -- so a name is region- and account-portable where an ARN pins both. Pinning the region is what forced the split in the first place: passing the us-west-2 ARN to a us-east-1 Nova job is rejected with "Model package group ARN region 'us-west-2' does not match expected region 'us-east-1'". One name serves both regions and drops a hardcoded account ID from a public repo. Verified: the bare name resolves to the same ARN via DescribeModelPackageGroup in us-west-2, and the us-west-2 recipe path still submits -- SFT, DPO and RLVR minimal-request tests pass against the service (3 passed). Collection unchanged at 100 tests. * test: resolve the shallow suite's CPU training image per region `CPU_IMAGE` hardcoded a us-west-2 URI in the public DLC account. Replace it with `cpu_image(sagemaker_session)`, which resolves the same image in the session's own region through `image_uris.retrieve` -- the resolver the SDK's framework estimators already use, so this is the supported mapping rather than a reconstruction of it. The registry account is not constant, which is what makes the hardcoded form actually wrong rather than merely untidy: it is 763104351884 across the commercial regions but 442386744353 in GovCloud and 727897471807 in China (on .com.cn). A pinned URI is unusable outside one partition, and it fails as an ECR error from the backend's role-assuming validators, which reads like a test bug rather than a hardcoded constant. A function rather than a constant because it needs the session's region; all three call sites already had a session in scope. Verified against AWS: reproduces the previously hardcoded URI byte-for-byte in us-west-2, and returns the correct in-region host (and per-partition registry) in us-east-1, eu-west-1, ap-northeast-1, us-gov-west-1 and cn-north-1. The affected tests pass on a real account -- 10 passed in 88s, covering the ModelTrainer helper, the raw TrainingJob.create path, and the tuner. * fix(train): bring the tuner path inside the shallow concurrency cap `_tuning()` submitted via `tuner.tune()` without acquiring slots, because a tuning job is stopped through `tuner.stop_tuning_job()` rather than `stop_quietly` and so never went through `submitted()`. Meanwhile the `DEFAULT_MAX_CONCURRENT_JOBS` note, the README quota table and `_requested_slots` all described the tuner as being inside the cap. It wasn't. Wrap it in `job_slots()` and drain after stopping. Slots are sized from the tuner's `max_parallel_jobs`, not a compute block: a tuning job occupies instance quota through the child training jobs it launches, which is also why `_requested_slots` cannot size this and `_tuning()` requests its own. The drain matters for the same reason it does elsewhere -- `stop_tuning_job()` returns while the job is still `Stopping` and its children are still tearing down, so releasing there is the release-before-terminal pattern that caused the ~37-concurrent breach. `_STATUS_ATTRS` already carried `hyper_parameter_tuning_job_status`, so the waiter handled this job type already; nothing ever called it with one. Renamed `_wait_until_terminal` -> `wait_until_terminal`. A test module outside the harness now needs it, and no other test imports a private name from there. Real impact today is small and worth saying so: both tuner tests are `max_parallel_jobs=1`, so this is 1 slot each. It is wired up because the cost is one context manager and the failure mode otherwise is silent -- a future test raising `max_parallel_jobs` would consume capacity outside a cap that still claimed to bound it. Verified: 3 unit scenarios (slots held through tune -> stop -> drain and released after; a failing test body still stops and releases; a missing or None `max_parallel_jobs` yields 1 slot, never an unbounded 0), plus a real run -- 2 passed in 28s, both jobs logging "reached Stopped; releasing slot" with no drain timeout. Docs corrected in all three places that overclaimed.jam-jee · 7198091d · 2026-08-19
- 2.1ETVAdd aws batch (#5409) * Add aws batch implementation (works with example notebook) * fixing unit tests and adding integration test * add example notebook * Adding missing dependencies for aws_batch * Fixing indentation bug in source code * comment out delete resources in example notebook * Add notebook png and remove extraneous comments * Add in png correctly * Removing logs_from_job from session_helper * Adding helpers for logging * Make helper methods internal * Adding back nest asyncio dependency * Updating unit tests for internal-external method changesaviruthen · f272de02 · 2025-12-19
- 1.6ETVBug fix for hmac key for V3 (#5379) * bug fix for hmac key and remove remote function from train * Remove remaining REMOTE_FUNCTION_SECRET_KEY references from tests * Add back remote function folder --------- Co-authored-by: Zhaoqi <52220743+zhaoqizqwang@users.noreply.github.com>aviruthen · fb0d789d · 2025-12-15
- 1.3ETVV3 Bug Fixes (#5601) * V3 Bug Fixes * fix(model_builder): Only set s3_upload_path for S3 URIs in passthrough In _build_for_passthrough(), model_path could be a local /tmp path. Setting s3_upload_path to a local path caused CreateModel API to reject the modelDataUrl with a validation error since it requires s3:// or https:// URIs. Now only S3 URIs are assigned to s3_upload_path; local paths are handled separately by _prepare_for_mode() in LOCAL_CONTAINER mode. * Test fixes * Bug fix 3 and 4Gokul Anantha Narayanan · 55a4ee59 · 2026-03-06
- 1.0ETVfix: Address Hyperparameter issue , validate s3 output path, additional unit tests (#5376) * fix: Fix the recipe selection for multiple recipe scenario * fix: Fix the recipe selection for multiple recipe scenario * fix: Hyperparameter issue fixes, validate s3 output path,additional unit tests --------- Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com>rsareddy0329 · 60574e56 · 2025-12-07