Syed Mujtaba

Nova release (#5969) * Add master-mtrl-trainer branch to PR checks * Add master-mtrl-release branch to PR checks * Add new branch for nova reconciliation in PR checks * wip: RLVRTrainer lambda arn support (#2025) * wip: RLVRTrainer lambda arn support * fix: add duplicate check in evaluator creation + added tests * fix: unit test bugs * feat: add optional hub_name param to AI registry class * fix bugs in new rlvr integ tests * fix: improve exception handling in finetune utils * fix: logging in _extract_evaluator_arn * fix: revert extra hub_name param in air_hub * fix: add gpu intensive marker for new integ tests * doc: add section in example notebook for passing lambda arn to RLVR Trainer --------- Co-authored-by: Syed Jafri <syedjfr@amazon.com> * feat: add is_multimodal utils function (#2033) * feat: add is_multimodal utils function * fix: Make method name Nova specific: nova_dataset_is_multimodal * fix: move to is_multimodal to data_utils file * fix: rename to is_multimodal_data * chore: ran code formatter * doc: make docstring nova agnostic * feat: type: Dataset support for is_multimodal method * feat: add is_multimodal param to model customization trainers * observability: add log statement for is_multimodal_data --------- Co-authored-by: Syed Jafri <syedjfr@amazon.com> * feat(train): Add 3-level recipe override support with get_resolved_recipe() (#2034) * Update pr-checks-master.yml * feat(train): Add 3-level recipe override support with get_resolved_recipe() Adds recipe file + programmatic overrides to trainers (SFT, RLVR, DPO), evaluators (BenchMarkEvaluator), and ModelTrainer. Users can now pass a YAML recipe file and/or overrides dict, then inspect the fully merged configuration via get_resolved_recipe() before submitting a job. Key additions: - RecipeResolver class: handles Hub template rendering, OmegaConf 3-way merge, validation against override-params spec, and caching - recipe/overrides params on SFTTrainer, RLVRTrainer, DPOTrainer, BenchMarkEvaluator constructors - get_resolved_recipe() on all trainers, evaluators, and ModelTrainer - 40 unit tests covering resolver logic and trainer integration - Example notebooks for SFT, evaluator, and ModelTrainer workflows * refactor: Address PE review - extract mixin, fix mutation bug, remove SSRF vector - Move get_resolved_recipe() to BaseTrainer (eliminates copy-paste across SFT/RLVR/DPO trainers) - Deep-copy user_dict before stripping protected keys (fixes mutation bug) - Remove HTTP/HTTPS URL support from recipe loading (SSRF prevention) - Clarify overrides docstring: introspection-only, not plumbed to train() - Remove unused import (from logging import exception) * feat(train): Add integration tests for recipe override, fix enum validation - Add integ tests for SFTTrainer and BenchMarkEvaluator get_resolved_recipe() (tests run against live Hub without submitting training jobs) - Fix enum validation to skip empty-string defaults (Hub specs sometimes have empty-string defaults that don't match their own enum constraints) - All 45 tests pass (30 unit + 10 trainer + 5 integ) * refactor: Move recipe support to BaseEvaluator, fix docstrings - Move recipe/overrides fields and get_resolved_recipe() from BenchMarkEvaluator to BaseEvaluator (now available to CustomScorer, LLMAsJudge, and all future evaluators) - Fix docstring: evaluators do 2-level merge (recipe < overrides), not 3-level (no Hub template for eval) - Fix ModelTrainer docstring: get_resolved_recipe() works for all recipe types, not just Nova - Remove duplicate get_resolved_recipe() from BenchMarkEvaluator * feat: Wire recipe into train()/evaluate() + add unit tests - SFT/RLVR/DPO train(): resolved recipe values now override matching keys in final_hyperparameters before job submission - Evaluators: _get_effective_hyperparameters() returns resolved recipe values when recipe/overrides are provided, used by evaluate() - Add 4 unit tests verifying recipe values flow into train()/evaluate(): - SFT train() applies recipe overrides to hyperparameters - SFT train() without recipe uses hyperparameters unchanged - Evaluator _get_effective_hyperparameters with recipe - Evaluator _get_effective_hyperparameters without recipe (fallback) - Fix comment: "overrides > recipe > Hub defaults" - Move recipe/overrides fields to BaseEvaluator for all evaluators * test: Add ModelTrainer integ tests for get_resolved_recipe() - test_model_trainer_get_resolved_recipe_with_local_yaml: verifies overrides win over recipe file values, recipe values preserved - test_model_trainer_get_resolved_recipe_overrides_only: verifies recipe-only path (no overrides) returns file values as-is - test_model_trainer_get_resolved_recipe_is_idempotent: verifies caching, deep-copy isolation from mutations * refactor: Extract flatten_resolved_recipe utility, DRY up trainers/evaluators - Add flatten_resolved_recipe() to recipe_resolver.py as shared utility - BaseTrainer._apply_recipe_to_hyperparameters() uses it (called by SFT/RLVR/DPO train()) - BaseEvaluator._get_effective_hyperparameters() uses it (called by BenchMark/CustomScorer evaluate()) - Eliminates duplicated flattening logic across all surfaces * feat: Add recipe/overrides support to MultiTurnRLTrainer and MultiTurnRLEvaluator - MultiTurnRLTrainer: add recipe/overrides params, wire into train() via _apply_recipe_to_hyperparameters() - MultiTurnRLEvaluator: uses _get_effective_hyperparameters() from BaseEvaluator (recipe values flow into template context) - Add 4 unit tests for MTRL trainer and evaluator recipe integration - Total: 56 tests passing * feat: Enable full recipe template override for non-spec keys Allow power users to override any key in the Hub recipe template, not just the UI-exposed spec subset. Fetches SmtjRecipeTemplateS3Uri at init time and uses it as the base layer for recipe resolution. - Add full_recipe_template param to RecipeResolver and resolve_recipe() - Fetch full recipe YAML in _get_fine_tuning_options_and_model_arn() - Remove filter in _apply_recipe_to_hyperparameters (all scalar keys flow through) - Skip nested dict/list values during hyperparameter flattening - Move in-function imports to file level - Add unit tests (TestFullRecipeTemplate, TestBuildKeyPathMap) - Add integration tests (TestSFTTrainerFullRecipeOverrideInteg) - Update notebook with power-user override example * Add branch for master-nova-reconciliation to workflow * fix: Deep-flatten nested recipe keys into hyperparameters flatten_resolved_recipe now recursively walks nested dicts to extract all scalar leaf values by key name. This fixes the issue where nested recipe fields like lr_scheduler.warmup_steps were dropped during _apply_recipe_to_hyperparameters because they were dict values. Users can now override nested keys: overrides={"training_config": {"lr_scheduler": {"warmup_steps": 30}}} and they correctly flow through as flat hyperparameters to the training job. * test: Add unit and integ tests for nested recipe key override flattening - Unit: test_deeply_nested_peft_keys_flow_into_hyperparameters - Unit: test_no_dicts_or_lists_in_final_hyperparameters - Integ: TestSFTTrainerNestedRecipeOverrideInteg (3 tests) - test_sft_nested_override_flows_to_hyperparameters - test_sft_nested_defaults_preserved_in_hyperparameters - test_sft_recipe_file_overrides_nested_keys * docs: Add nested recipe attribute example to notebook Demonstrates how deeply nested recipe keys (lr_scheduler.warmup_steps, peft.rank) are handled: overridden via recipe_overrides and auto-flattened into scalar hyperparameters for the training API. * feat: Add Nova SMI config bounds validation to ModelBuilder (#2040) * feat: Add Nova SMI config bounds validation to ModelBuilder Validates user-provided CONTEXT_LENGTH and MAX_CONCURRENCY against supported Nova inference tier limits before deployment. Previously, invalid configurations would only fail at container runtime with cryptic errors. Changes: - Add _NOVA_SMI_TIERS table with per-(model, instance) tier bounds - Add _validate_nova_smi_config() that checks env vars against tiers - Fix env var merge order so user overrides take priority over defaults - Wire validation into both Nova deploy paths - Add unit tests covering valid configs, boundary violations, and skip cases * refactor: Use _NOVA_HOSTING_CONFIGS for SMI validation instead of separate tier map Remove the duplicate _NOVA_SMI_TIERS map and rewrite _validate_nova_smi_config() to validate against the existing _NOVA_HOSTING_CONFIGS. This avoids maintaining two separate maps that must stay in sync. * refactor: Add Tiers to _NOVA_HOSTING_CONFIGS for accurate per-tier validation Embed full (max_context_length, max_concurrency) tier bounds directly into each _NOVA_HOSTING_CONFIGS entry. This removes the need for a separate _NOVA_SMI_TIERS map while preserving accurate intermediate-tier validation (e.g., at 64k context on p5, max concurrency is 32 not 8). * docs: Add env_vars configuration example for Nova deployment Add a section to the deployment notebook showing how to set CONTEXT_LENGTH and MAX_CONCURRENCY when deploying Nova models, including the tier bounds table and an example of validation error output. * fix: Add missing (256000, 2) tier for NOVA_LITE_2 on ml.p5.48xlarge Sync with Forge SDK SUPPORTED_SMI_CONFIGS which includes a 256k context tier that was missing from our config. * Feat/inspect ai evaluator (#2039) * feat: Support InspectAI evaluation. * Add InspectAI evaluation demo notebook * address black and isort * address flake8 * fix: accept trainer as model param, make region us agnostic --------- Co-authored-by: Luke Luneau <luluneau@amazon.com> * Iam role creation (#2041) * feat: Add Nova SMI config bounds validation to ModelBuilder Validates user-provided CONTEXT_LENGTH and MAX_CONCURRENCY against supported Nova inference tier limits before deployment. Previously, invalid configurations would only fail at container runtime with cryptic errors. Changes: - Add _NOVA_SMI_TIERS table with per-(model, instance) tier bounds - Add _validate_nova_smi_config() that checks env vars against tiers - Fix env var merge order so user overrides take priority over defaults - Wire validation into both Nova deploy paths - Add unit tests covering valid configs, boundary violations, and skip cases * refactor: Use _NOVA_HOSTING_CONFIGS for SMI validation instead of separate tier map Remove the duplicate _NOVA_SMI_TIERS map and rewrite _validate_nova_smi_config() to validate against the existing _NOVA_HOSTING_CONFIGS. This avoids maintaining two separate maps that must stay in sync. * refactor: Add Tiers to _NOVA_HOSTING_CONFIGS for accurate per-tier validation Embed full (max_context_length, max_concurrency) tier bounds directly into each _NOVA_HOSTING_CONFIGS entry. This removes the need for a separate _NOVA_SMI_TIERS map while preserving accurate intermediate-tier validation (e.g., at 64k context on p5, max concurrency is 32 not 8). * docs: Add env_vars configuration example for Nova deployment Add a section to the deployment notebook showing how to set CONTEXT_LENGTH and MAX_CONCURRENCY when deploying Nova models, including the tier bounds table and an example of validation error output. * fix: Add missing (256000, 2) tier for NOVA_LITE_2 on ml.p5.48xlarge Sync with Forge SDK SUPPORTED_SMI_CONFIGS which includes a 256k context tier that was missing from our config. * docs: Add IAM role auto-creation interfaces notebook Demonstrates the resolve_or_create_role auto-resolution behavior across the trainer, evaluator, model builder, pipeline, and feature store entry points. * feat: Auto-create least-privilege IAM roles when none provided Adds resolve_or_create_role() in sagemaker-core that resolves an existing role (explicit, caller identity, or default auto-role) via permission simulation and only creates a new least-privilege role when none is sufficient. Idempotent get-or-create with policy config in iam_policies.json (shipped as package-data). Wires it into the training, evaluation, serving, pipeline, and feature store entry points, replacing get_execution_role() fallbacks and the pipeline 'IAM role is required' ValueError. Updates affected unit tests and adds resolver, serving, and feature_store coverage. Known follow-up: feature_scheduler.put_trigger() still uses get_execution_role (EventBridge Rule needs events.amazonaws.com trust). * refactor: inline IAM policy config as Python constant; relocate example notebook - Convert iam_policies.json to a Python constant (IAM_POLICY_CONFIG in iam_policies.py). Source modules are always packaged, so this removes the need for the helper/*.json packaging references that would otherwise be required for the resolver to find the config at runtime in installed wheels. - Drop the now-unnecessary 'recursive-include src/sagemaker/core/helper *.json' (MANIFEST.in) and 'helper/*.json' package-data (pyproject.toml). - Resolver imports the constant instead of reading the bundled JSON via importlib.resources. - Move notebooks/iam_role_autocreate_interfaces_real.ipynb to v3-examples/iam_role_autocreation.ipynb. * refactor: Hoist IAM resolver imports to module top, document role resolution order Moves the resolve_or_create_role imports from inline (function-body) to the top of each consuming module. Adds comments at each call site documenting the resolution order: explicit role > caller session role (if sufficient) > on-demand least-privilege role. Updates test patch targets to patch resolve_or_create_role in each consumer's namespace (where it is now looked up) rather than the source module. * test: Mock resolve_or_create_role in train/dataset tests after IAM auto-create The IAM role auto-creation change (23f8c9b4) rewired TrainDefaults.get_role() to call resolve_or_create_role(), which issues boto STS calls and subscripts the responses. Tests that build a ModelTrainer/DataSet without an explicit role now hit that path with spec'd/bare mocks and fail. Patch resolve_or_create_role (matching the test_defaults.py pattern): - model_trainer: patch in the shared autouse modules_session fixture - dataset: add decorator to the two S3-location create tests * feat(iam): address PR review on role auto-creation Addresses Vaishnavi's review comments on PR #2041: - Policy drift: _ensure_policies_attached now syncs an existing policy's document via CreatePolicyVersion (pruning oldest version at the 5-version limit) when it differs from the current definition, instead of skipping. - Bedrock: add a 'bedrock' role_type (bedrock.amazonaws.com trust + S3/KMS least-privilege) and auto-resolve it in BedrockModelBuilder.deploy() for both Nova and OSS paths, replacing the 'role_arn is required' ValueError. - Model package: grant sagemaker:AccessModelPackage (scoped to model-package/*) on the bedrock and serving roles for consuming restricted model packages. - Least-privilege: scope the pipeline sagemaker_policy from Resource '*' to the specific SageMaker resource-type ARNs, and add the required sagemaker:AddTags dependent action. Tests: resolver (policy sync, bedrock role type) and bedrock builder (auto-resolve on Nova/OSS) coverage added/updated. * feat(iam): address second round of PR review on role auto-creation Addresses follow-up review comments on PR #2041: - S3/KMS scoping now accept a single value or a list (Union[str, List[str]]), defaulting to '*'. Each bucket expands to bucket + object ARNs; each KMS key to a key ARN. A list containing '*' collapses to the wildcard. - KMS ARNs are scoped to the caller's account (arn:<p>:kms:*:<account>:key/<id>); region is left '*' to allow cross-region key references. - iam:PassRole resource scoped from '*' to arn:<p>:iam::<account>:role/* (via an IAM_PASSROLE_PLACEHOLDER), keeping the PassedToService=sagemaker condition. - Session fallback uses sagemaker core Session() instead of boto3 directly, so it inherits SDK config/region defaults; dropped the now-unused boto3 import. Tests: list-based S3/KMS expansion, wildcard collapse, account-scoped KMS and PassRole, and the boto-session fallback branches. * feat(iam): log auto-created role and attached/updated policies at WARNING Addresses PR #2041 review: surface account-mutating actions so the user is not taken by surprise. The new IAM role and any policies created, updated, or attached on their behalf are now logged at WARNING (visible by default), while no-op cases (role reuse, already-current policies) stay at INFO. Adds caplog tests for the create/update/no-op paths and the role-creation warning. * feat(train): add Serverless/SMTJ/HyperPod support to trainers and evaluators (#2045) * Add branch for master-nova-reconciliation to workflow * feat(train): add multi-compute backend support (Serverless/SMTJ/HyperPod) to trainers and evaluators * fix gitignore * Addreee PR comments * Restore Compute class name * Update benchmark evaluator * Remove duplicate code * Move training_image out of Compute class * Remove invoke_batch from ModelBuilder * Trainer class fixes * Fixed Trainers and Evaluators; Added CPT Trainer * Clean up comment --------- Co-authored-by: Gokul Anantha Narayanan <166456257+nargokul@users.noreply.github.com> * RLVR reward lambda validation (#2036) * initial commit: copy verifier from nova sdk * feat: rename RFT to RLVR + improve lambda arn checking * feat: update reward verifier + add tests * fix: check lambda arn has "sagemaker" only for nova models * feat: add unit tests + remove sample dataset validation (to be handled by DataPrep APIs) * doc: add example notebook * cleanup: remove misleading unit test * fix: ensure validator is called in rlvr_trainer * feat: Use compute classes in rlvr lambda verification * fix: update rlvr tests to use compute param --------- Co-authored-by: Syed Jafri <syedjfr@amazon.com> * feat: Adding infra validation (#2049) * Add branch for master-nova-reconciliation to workflow * feat: Adding infra validation * fix: make sagemaker session optional param --------- Co-authored-by: Gokul Anantha Narayanan <166456257+nargokul@users.noreply.github.com> Co-authored-by: Syed Jafri <syedjfr@amazon.com> * fix: add hyperpod validation in train and evaluate (#2051) * feat(train): auto-resolve HyperPod recipe from Hub (#2050) * feat(train): auto-resolve HyperPod recipe from Hub Remove the recipe field from HyperPodCompute class since the Trainer and Evaluator already accept recipe as a direct parameter. When recipe is not provided, the SDK now auto-resolves it from SageMaker Hub metadata * update recipe fetching fallback * fix: harden auto-created IAM role security - Add aws:SourceAccount condition to all five auto-role trust policies, scoped to the caller's account at create time, to prevent the cross-service confused-deputy problem. - Narrow iam:PassRole from role/* to role/SageMaker-AutoRole-*, so the auto pipeline role can only pass the SDK's own least-privilege roles. - Tag auto-created roles (CreatedBy, sagemaker:auto-created-role, RoleType) on creation, and backfill tags on reused/pre-existing roles. Add iam:TagRole to the required actions. - Add unit tests covering account scoping, PassRole narrowing, and tagging. * Revert "fix: harden auto-created IAM role security" This reverts commit bb0af9127e7676eae956cbd64c77d5601434d9ac. X-AI-Prompt: Revert the directly-pushed IAM hardening commit from the launch branch so it can be reintroduced via PR X-AI-Tool: kiro-cli * fix: recipe override handling (#2056) * fix: recipe override handling * fix: SMTJ serverful bugs in recipe validation * feat: add Nova as target for LLMAJ (#2059) * wip: add nova specific recipe validations (#2052) * wip: add nova specific recipe validations * feat: Ensure nova specific recipe validations in BaseTrainer * fix: skip nova replicas validation for serverless smtj * cleanup: skip instance type validation for smtj serverless * fix: combine both nova and non-nova validations * fix: integ test bugs * fix: address review comments - renaming variables, updating params etc. * fix: removed replicas check --------- Co-authored-by: Syed Jafri <syedjfr@amazon.com> * feat(train): enable Data Mixing for Nova models#2047 (#2054) * fix: use compute param in get fine tuning utils (#2060) * Hyperpod iam creation (#2057) * feat(iam): add HyperPod IAM support — job-role type and caller-side CLI permission checks Extends the IAM role auto-creation from #2041 to cover the HyperPod training/evaluation path added in #2045. HyperPod has two distinct identities, which this change keeps separate: - Job execution role (assumed by the job ON the cluster, trusted by sagemaker.amazonaws.com): a new "hyperpod" role type carries only job-runtime permissions (S3/ECR/CloudWatch/KMS). It omits the VPC-mode EC2 ENI block, which does not apply to HyperPod cluster jobs. Provision it via TrainDefaults.get_role(role_type="hyperpod"). - Caller identity (runs the HyperPod CLI connect-cluster + start-job locally under its own credentials): needs sagemaker:DescribeCluster, eks:DescribeCluster, eks:AccessKubernetesApi. The SDK cannot auto-create a role for this, so verify_hyperpod_connect_permissions() simulates these against the caller and logs a non-blocking WARNING when any are missing. Wired into both HyperPod submit paths (BaseTrainer._train_hyperpod and BaseEvaluator._submit_hyperpod_eval_job). These are deliberately NOT attached to the SageMaker-trusted job role, which the local caller cannot assume. Also: - _resolve_explicit_role() now accepts role ARNs from any partition (aws-cn/aws-us-gov), not just commercial "aws". - _raise_auto_creation_error() derives the required trust principal from the role config instead of hardcoding sagemaker.amazonaws.com. Tests: - Unit tests for the hyperpod job role (incl. asserting it excludes the CLI connect permissions), the connect-permission verifier, partition handling, and role_type forwarding. - A no-mocks end-to-end integ test that creates the role against real IAM, asserts the job role excludes the connect permissions, checks idempotency, and self-cleans. Verified end-to-end against a live account. * test(iam): add HyperPod connect-permission unit tests + example notebook - Add tests/unit/train/test_hyperpod_connect_permissions.py covering BaseTrainer._train_hyperpod: it verifies the caller's HyperPod CLI connect permissions (forwarding cluster_name), the missing-cluster_name guard fires before that check, and a False/None verdict warns without blocking submit. - Fix a latent bug surfaced by the tests: validate_hyperpod_compute was called in _train_hyperpod but never imported (would NameError on any real run). Added the local import, matching how the evaluator path imports it. - Add v3-examples/hyperpod_iam_role.ipynb: a no-mocks example showing the two HyperPod identities — auto-creating the SageMaker-AutoRole-HyperPod job role, confirming it excludes the CLI connect permissions, and running the caller-side verify_hyperpod_connect_permissions check. Self-cleaning. * refactor(iam): unify compute permission checks behind an infra_type dispatcher Addresses review feedback on the HyperPod IAM changes: 1. Reuse: extract _simulate_denied_actions() in the resolver so both _role_has_sufficient_permissions and verify_hyperpod_connect_permissions share one paginated iam:SimulatePrincipalPolicy implementation instead of duplicating the loop. 2. Imports: hoist the previously-local imports (verify_hyperpod_connect_permissions, validate_hyperpod_compute, _is_nova_model, TrainDefaults) to module top in base_trainer and base_evaluator. No circular-import constraint forced them local. 3+4. Single dispatcher keyed on infra type: add TrainDefaults.get_infra_type(compute) ({serverless, training_job, hyperpod}) and TrainDefaults.resolve_compute_permissions (infra_type, ...). HyperPod verifies caller-side CLI connect permissions and resolves no role; serverless/training_job resolve the training execution role. _train_hyperpod and _submit_hyperpod_eval_job now call this one entry point instead of inlining the split logic. (Per guidance, the shipped resolve_or_create_role(role_type=...) API is left unchanged.) Reverts the temporary role_type param previously added to TrainDefaults.get_role. Tests updated: resolver tests cover _simulate_denied_actions; defaults tests cover get_infra_type + resolve_compute_permissions dispatch; the hyperpod connect tests patch the new dispatcher seam. * fix(notebook): use resolve_or_create_role for HyperPod job role The refactor removed the role_type param from TrainDefaults.get_role, so the notebook's get_role(role_type="hyperpod") call would TypeError. Call resolve_or_create_role(role_type="hyperpod") directly (as the integ test does) and correct the accompanying prose. * refactor(iam): address review — drop dispatcher, partition-aware ARNs, e2e test - Remove get_infra_type and resolve_compute_permissions (the dispatcher was dead code: get_infra_type was never called and the polymorphic ARN-or-None return was a foot-gun). Replace with the intent-revealing TrainDefaults.verify_hyperpod_caller_permissions, which the HyperPod trainer and evaluator paths call directly; serverless/SMTJ continue to use get_role. - Make static policy ARNs partition-aware: _replace_placeholders now rewrites literal arn:aws: resources (e.g. the hyperpod cloudwatch_logs_policy) to the caller's partition, so GovCloud/China/ISO get a matching ARN. No-op for "aws". - Add an end-to-end test (TestTrainHyperPodConnectPermissionsEndToEnd) that runs base_trainer -> real verify helper -> real resolver internals with only the boto IAM/STS clients mocked: a denied connect action warns without blocking submit, and an all-allowed caller produces no warning. Tests updated: resolver tests cover the partition rewrite; defaults tests cover verify_hyperpod_caller_permissions; connect tests patch the new seam. * fix: harden auto-created IAM roles and protect curated recipe keys (#2058) Strengthen the IAM roles the SDK auto-creates in the customer's account and add tamper protection to the training recipe resolver: - Add an aws:SourceAccount condition to all auto-role trust policies, scoped to the caller's account, to prevent the cross-service confused-deputy problem. - Narrow iam:PassRole from role/* to role/SageMaker-AutoRole-*, so the auto pipeline role can only pass the SDK's own least-privilege roles. - Tag auto-created roles (CreatedBy) on creation and backfill on reused roles for attribution and safe cleanup; add iam:TagRole. - Warn when an auto-created role is granted account-wide ("*") S3, KMS, or Glue catalog access, and point to the s3_resource/kms_resource parameters (or supplying an explicit role) to scope it down. - Protect dataset_catalog (alongside model_type and model_name_or_path) from being overridden by user recipes/overrides. - Add unit tests covering all of the above. X-AI-Prompt: Resolve rebase conflicts and squash the IAM role hardening and recipe-key protection changes onto the latest target branch X-AI-Tool: kiro-cli * Recipe validation - unit and integ tests (#2062) * testing: add integ tests for recipe validation * feat: added unit tests for recipe validation * fix: add replicas validation with HyperpodCompute --------- Co-authored-by: Syed Jafri <syedjfr@amazon.com> * fix: set Converse as S3DataType for Nova models in SMTJ Serverful for SFT and DPO (#2064) * fix: hub-content IAM perms, recipe dataset paths, and log markup escaping (#2065) * Add branch for master-nova-reconciliation to workflow * fix: add hub-content IAM perms, inject recipe dataset paths, disable log markup Three independent fine-tuning fixes, each with unit coverage. 1. IAM — DescribeHubContent for fine-tuning roles (sagemaker-core/src/sagemaker/core/helper/iam_policies.py) Fine-tuning jobs reference a base model via a hub-content ARN (e.g. SageMakerPublicHub/Model/<name>/<version>). The SageMaker service reads that hub content as the execution role when it creates the training job, so the role needs sagemaker:DescribeHubContent. Without it, CreateTrainingJob fails with "Access denied to hub content". Added a scoped hub_content_policy (arn:aws:sagemaker:*:*:hub-content/*) to both the "training" and "hyperpod" roles. The account segment is left wildcard so the policy covers the public hub (the "aws" account) and private hubs (SAGEMAKER_HUB_NAME). 2. Recipe — inject resolved dataset channel paths (sagemaker-train/src/sagemaker/train/base_trainer.py) _train_serverful_smtj rendered the recipe template with empty data.train_files / data.val_files, so the training container aborted. Now the resolved training/validation datasets are mapped to their container channel mount paths (/opt/ml/input/data/<channel>) and injected into the recipe override spec before rendering. An S3 prefix maps to the channel directory; a trailing object key with an extension maps to the mounted file. Datasets are resolved once up front (the later resolution was removed) and the trainer environment is now forwarded to ModelTrainer.from_recipe. 3. Logs — disable rich markup for streamed container log lines (sagemaker-core/src/sagemaker/core/tools/templates.py + regenerated sagemaker-core/src/sagemaker/core/resources.py) Container log lines are arbitrary text and may contain square brackets (e.g. file paths like [.../main_ppo.py]). The rich logging handler parsed them as markup tags and raised MarkupError, aborting the wait loop. The PRINT_WAIT_LOGS codegen template now passes extra={"markup": False} when logging streamed events; resources.py was regenerated via codegen so the change lands in TrainingJob, ProcessingJob, and TransformJob wait() loops. The generated file was NOT hand-edited — the template is the source of truth. Tests - test_iam_role_resolver.py: training and hyperpod roles carry DescribeHubContent scoped to hub-content/*. - test_base_trainer_serverful.py (new): S3 prefix vs object-key channel mount mapping, validation-absent case, in-place spec mutation, and environment forwarding. - test_resources.py (TestWaitLogMarkup): TrainingJob/ProcessingJob/ TransformJob wait() log streamed lines with extra={"markup": False}. * fix: Evaluation on hyperpod (#2061) * fix: Evaluation on hyperpod Fixed E2E Benchmark and custom evaluation experience using trained models on hyperpod * fix endpoint deployment * bug fixes in evaluator validation * address PR comments * fix: skip model_package_group validation when HyperPod compute is provided in CPTTrainer (#2068) _validate_and_resolve_model_package_group() does not accept a 'compute' kwarg. Guard the call with 'if compute is None' to match the pattern used by SFT, RLVR, and DPO trainers. * fix: resolve HyperPod training image from EKS payload template (#2069) For HyperPod-only techniques like CPT (non-datamixing), get_training_image() looks at SmtjImageUri which is not populated for HyperPod recipes. This caused jobs to be submitted without a container image, leading to PyTorchJob validation failures. Updated _train_hyperpod image resolution: 1. User-provided training_image 2. get_training_image() (SmtjImageUri) with SM-TJ -> SM-HP tag swap 3. Fallback: get_hyperpod_training_image() extracts image from template 4. Raise ValueError if still unresolved Added extract_image_from_hyperpod_template() as shared helper, refactored data_mixing_utils to use it. Added unit tests. * fix: apply recipe overrides to hyperparameters in SMTJ serverful path (#2070) The _train_serverful_smtj code path was not calling _apply_recipe_to_hyperparameters(), so user-provided overrides (e.g. overrides={'max_epochs': 1, 'name': '...'}) were silently dropped from the hyperparameters dict passed to ModelTrainer.from_recipe(). Added the same _apply_recipe_to_hyperparameters() call that the serverless path already uses, placed after subclass extra HP injection. Unit tests added to verify overrides overwrite base HP values and coexist with subclass-injected extra hyperparameters. * feat: support serverful training job checkpoint resolution in InspectAI evaluator (#2066) * fix: reject unknown recipe overrides (serverless + serverful) and untrusted IAM roles (#2071) * fix: drop unknown recipe overrides and reject untrusted IAM roles Two bug-bash fixes for the Nova reconciliation flow. 1. Recipe overrides (SFT and other trainers) Override keys that don't exist in the base recipe were silently merged into the resolved recipe. Now RecipeResolver drops any override/user-recipe key with no counterpart in the base recipe (recursing into nested mappings, preserving known siblings) and logs a warning naming the dropped key and its source. Power-user overrides of non-spec keys that DO exist in the recipe still apply. 2. IAM role resolution (auto-create fallback) resolve_or_create_role reused the caller's role whenever it passed the permission simulation, without checking the role's trust policy. An admin role with all permissions but no sagemaker.amazonaws.com trust was handed to CreateTrainingJob, which failed with "Could not assume role". The resolver now also verifies the role trusts the required service principal; on a definitive "not trusted" it warns and falls back to the auto-created role. Adds unit tests covering both behaviors. * fix: drop unknown recipe overrides on serverful SMTJ path Bug 3: on the serverful SMTJ recipe path (ModelTrainer.from_recipe -> _load_base_recipe), override keys that don't exist in the base recipe (e.g. max_steps for a model whose recipe has no such field) were merged in unvalidated with no warning. _load_base_recipe now filters recipe_overrides against the loaded base recipe before the OmegaConf merge: any key (top-level or nested) with no counterpart in the recipe is dropped and a warning naming it is logged, while valid overrides and known siblings are preserved. This mirrors the drop-and-warn behavior already applied in RecipeResolver for the serverless path (bug 18). Adds unit tests for _load_base_recipe and the _drop_unknown_recipe_overrides helper. * test: use placeholder account and generic role name in IAM tests Address PR feedback: replace the real AWS account 618100645563 with the documentation placeholder 123456789012, and the Amazon-specific role name IibsAdminAccess-DO-NOT-DELETE with a generic 'Admin' role in the IAM role resolver unit tests. * Fix recipe override errors in evaluator (#2067) * fix: Evaluation on hyperpod Fixed E2E Benchmark and custom evaluation experience using trained models on hyperpod * fix endpoint deployment * bug fixes in evaluator validation * address PR comments * fix: custom scorer evaluators Fix recipe override handling * extend evaluators to support OSS models * Fix custom eval for OSS model * Fix custom eval and add unit tests * fix unit tests failures * fix: MLFlow error causing OSS model eval to fail (#2075) * fix: Evaluation on hyperpod Fixed E2E Benchmark and custom evaluation experience using trained models on hyperpod * fix endpoint deployment * bug fixes in evaluator validation * address PR comments * fix: custom scorer evaluators Fix recipe override handling * extend evaluators to support OSS models * Fix custom eval for OSS model * Fix custom eval and add unit tests * fix unit tests failures * fix: mlflow key name error causing non-Nova eval to fail * update mlflow fix * Add mlflow flields to SMTJ serverful * fix(train): use Converse S3DataType for Nova SFT/DPO in serverless flow (#2079) The serverless training path hardcoded s3_data_type="S3Prefix" for all models in _convert_input_data_to_channels. For Nova SFT/DPO multimodal datasets this must be "Converse" so the SageMaker data agent downloads images and rewrites "uri" to "localPath" in the JSONL. Without this, the container validator fails with "Invalid S3Location, 'uri' is required". Add s3_data_type parameter to _convert_input_data_to_channels (default "S3Prefix" for backward compat) and pass "Converse" from sft_trainer and dpo_trainer when the model is Nova — matching the serverful SMTJ fix. * [Bugfix] RLVR Setup and Reward Lambda handling (#2076) Co-authored-by: Ealynn Hsu <ealynnh@amazon.com> * fix(train): route .hyperparameters.* through recipe resolver (#2078) * fix(train): route .hyperparameters.* through recipe resolver on serverless Remove early-return guard in _apply_recipe_to_hyperparameters so user-set values reach get_resolved_recipe_from_context, which promotes them into synthetic overrides the service will honor. * Update Base Evaluator * fix resolved recipes * Fix: Check s3 permission before saving recipe.yaml (#2083) * Fix: Check s3 permission before saving recipe.yaml * Adding unit tests * Address comments --------- Co-authored-by: Ealynn Hsu <ealynnh@amazon.com> * Bugfix: get_resolve_recipe() includes all overrides and displays as a nested dictionary (#2084) * WIP: get_resolved_recipe structure fixes * WIP: recipe resolution * patch output and data s3 paths in recipe preview * Unit tests and adding the logger properly --------- Co-authored-by: Ealynn Hsu <ealynnh@amazon.com> * fix(train): validate recipe and instance count override for SMTJ serverful (#2082) Co-authored-by: Mahima Chaudhary <mahchy@amazon.com> * fix(iam): validate roles by default, opt-in creation, and add MLflow perms (#2080) * fix(iam): validate roles by default, opt-in creation, and add MLflow perms Security hardening of IAM role handling plus MLflow support for training/eval. Validate-by-default (no silent auto-creation): - resolve_or_create_role is replaced by read-only resolve_and_validate_role, which resolves the provided/caller role and validates permissions + trust via iam:SimulatePrincipalPolicy. It never creates or mutates IAM. A missing permission raises RoleValidationError listing the exact missing actions and remediation; an unverifiable verdict (caller lacks SimulatePrincipalPolicy) warns and proceeds so Studio/notebook flows are not broken. - Explicit, opt-in creation moves to the IamRoleResolver class (iam_role_creator.py): create_execution_role / delete_execution_role / get_required_actions. This is the only code path that writes IAM. - All call sites (training defaults, evaluator, ModelBuilder, Bedrock, Pipeline, feature_store scheduler) switched to validate-only. MLflow permissions: - Add mlflow_policy to the training role (covers eval, which resolves role_type="training"). Enumerated sagemaker-mlflow data-plane actions only (no AccessUI, no destructive/control-plane actions), scoped to mlflow-app/* and mlflow-tracking-server/*. Training/eval containers log to managed MLflow as the execution role via the sagemaker-mlflow SigV4 plugin. Review fixes: - Validation gates only on *-resource "smoke test" actions; resource-scoped actions (mlflow/hub/model-package) are excluded so simulating without ResourceArns can't produce false denials (and non-MLflow jobs aren't blocked). - Add the aws:SourceAccount confused-deputy condition to the hyperpod trust policy (was the only role missing it). Tests + docs: reworked unit tests (validate-only + creator), new integ test for the validation path, and an iam_role_validation.ipynb example; deprecation banners on the old auto-creation notebooks. * fix(iam): address PR feedback — ListHubs, full MLflow set, validate provided roles - Add sagemaker:ListHubs to the hub_content_policy (training + hyperpod) so the hub-content resolution path can list hubs. - Expand mlflow_policy to mirror the Nova Forge SDK's documented MLflowSageMaker policy (nova-customization-sdk/docs/iam_setup.md): adds the experiment-tracking, logged-model, and model-registry actions (Create/Update RegisteredModel + ModelVersion, FinalizeLoggedModel, LogInputs/Outputs, Get*/Search*, AccessUI, Restore*, tag actions). Nothing removed; still scoped to mlflow-app/* and mlflow-tracking-server/*. - Validate user-PROVIDED roles, not just the caller fallback: ModelBuilder (build/deploy paths), Bedrock, and Pipeline now pass the user's role into resolve_and_validate_role instead of only calling it when no role was given. Construction-time ModelBuilder still only resolves a default when none is supplied (no IAM work when a role is provided). Train + feature_scheduler already threaded the provided role. - Drop the "never auto-created" phrasing from comments/docstrings/error text. - Update affected unit tests (serve, mlops) to stub resolve_and_validate_role on the now-validated provided-role paths. Fix a stray typo in the example notebook. All affected unit suites pass: core 76, train 233, serve 126, mlops 69. * fix(iam): address PR review — merge creator into resolver, MLflow describe, cleanups - Merge iam_role_creator.py into iam_role_resolver.py: the IamRoleResolver class (create/delete/get_required_actions) now lives alongside the read-only resolve_and_validate_role in the resolver module, so the class name matches the file. Deleted iam_role_creator.py; helper __init__ exports IamRoleResolver from the resolver module. (review: class name should match file) - mlflow_policy: add a control-plane describe statement (sagemaker:DescribeMlflowApp + DescribeMlflowTrackingServer) scoped to mlflow-app/* and mlflow-tracking-server/*, so a job can resolve the tracking endpoint from a provided MLflow ARN. (review: is DescribeMlflowTrackingServer needed when a user provides an MLflow app — yes) - create_execution_role: call get_caller_identity once. (review nit) - Delete the deprecated example notebooks iam_role_autocreation.ipynb and hyperpod_iam_role.ipynb (auto-creation removed; iam_role_validation.ipynb replaces them). (review: are these still in use) IAM unit suite: 76 passed. * fix(iam): add MLflow permissions to the hyperpod execution role MLflow applies to HyperPod jobs: when an mlflow_resource_arn is provided, the tracking URI/experiment/run are injected into the HyperPod recipe (base_trainer._train_hyperpod), and the job logs to managed MLflow on the cluster as its execution role — same as the SMTJ/serverless training path. The hyperpod role config was missing mlflow_policy, so add it (data-plane sagemaker-mlflow logging + control-plane DescribeMlflowApp/DescribeMlflowTrackingServer), mirroring the training role and scoped to mlflow-app/* and mlflow-tracking-server/*. Tests updated: assert MLflow perms on both training and hyperpod (the execution-role types that log to MLflow) and absent elsewhere. 77 passed. * fix(iam): add lambda_policy to hyperpod role (RLVR reward Lambda parity) Audit of training vs hyperpod execution-role policies found one more gap beyond MLflow: RLVR supports a Lambda custom_reward_function and runs on HyperPod (HyperPodCompute -> _train_hyperpod). When the reward function is a Lambda ARN, the job invokes it as the execution role, so the hyperpod role needs lambda:InvokeFunction — which only the training role had. Added it (mirrors training; function:* resource). Other training/hyperpod diffs were reviewed and are intentional: model_package (HyperPod doesn't register model packages), ec2 ENI (VPC-mode SMTJ only), and the cloudwatch_logs resource prefix (HyperPod logs aren't under /TrainingJobs*). 78 unit tests passed. * fix(iam): grant evaluation lineage + training-job permissions to the training role The evaluation pipeline runs as the training execution role (SageMaker-AutoRole- Training) and failed two steps: - CreateEvaluationAction: missing sagemaker:CreateAction on action/* - EvaluateCustomModel: missing sagemaker:AddTags on training-job/* (the existing model_package AddTags was scoped to model-package* resources only) Add an evaluation_policy to the training role covering the lineage operations the eval flow performs (Action/Artifact/Context create+describe+associations, scoped to action/*, artifact/*, context/*) and the training-job launch+tag (CreateTrainingJob/DescribeTrainingJob/StopTrainingJob/AddTags on training-job/*). 79 unit tests passed. * feat(train): add iterative training with base_model_name param (#2085) Add support for iterative training (resuming from S3 checkpoints) Co-authored-by: Mahima Chaudhary <mahchy@amazon.com> * Address PR #2074 review comments: codegen-emitted model_artifacts synthesis + tests (#2086) * feat: add support for Bedrock deployments from S3 paths * Address PR #2074 review comments - Emit TrainingJob.get() model_artifacts synthesis from the codegen engine (resources_codegen._get_get_method_post_processing + GET_METHOD_TEMPLATE {post_processing} hook) so the customization survives the next autogeneration of resources.py. - Add unit tests for the TrainingJob.get() model_artifacts synthesis (TestTrainingJobGetModelArtifactsSynthesis). - Add regression tests for trainer-checkpoint resolution from _latest_training_job.model_artifacts in BaseEvaluator._submit_hyperpod_eval_job and InspectAIEvaluator._resolve_trainer_model. * Fix hardcoded service-2.json path in data_extractor; regenerate resources.py - Remove the hardcoded SERVICE_JSON_FILE_PATH override pointing at a foreign developer's machine; fall back to the package-relative path already imported from constants (_PACKAGE_ROOT-based), so codegen runs from any clone. - Regenerate resources.py: the TrainingJob.get() model_artifacts synthesis block is emitted identically by the codegen engine; black (pinned 24.3.0) collapses the ModelArtifacts(...) call to one line. Confirmed idempotent and that config_schema.py / shapes.py regenerate with no changes. * Rename _get_get_method_post_processing to _get_method_post_processing Address review comment: drop the doubled 'get_get' in the codegen helper name. Updates the definition, call site, and the test-file docstring reference. --------- Co-authored-by: Lisa Ni <nilis@amazon.com> * release: update version and changelogs * fix: update changelog * fix: simplify subscription recipe handling and fix unit tests (#5973) Remove duplicate {customer_id} replacement and redundant subscription recipe override_params merge logic. Update tests to mock resolve_and_validate_role and get_hyperpod_recipe_path introduced in recent IAM/recipe validation changes. Co-authored-by: Vaishnavi Kommaraju <vvak@amazon.com> * fix: remove unused training_image tests (#5972) * fix: add training_image param to Compute class * fix: remove non-existing training image field from Compute unit tests --------- Co-authored-by: Syed Jafri <syedjfr@amazon.com> * Update documentation and unit tests (#5971) * fix: recipe unit tests to use jumptstart tags (#5974) * fix: add training_image param to Compute class * fix: remove non-existing training image field from Compute unit tests * fix: use get_jumpstart_tags in unit tests --------- Co-authored-by: Syed Jafri <syedjfr@amazon.com> --------- Co-authored-by: Gokul Anantha Narayanan <166456257+nargokul@users.noreply.github.com> Co-authored-by: Syed Jafri <syedjfr@amazon.com> Co-authored-by: LN <133025223+amazeAmazing@users.noreply.github.com> Co-authored-by: Luke Luneau <luluneau@amazon.com> Co-authored-by: Zhaoqi <jzhaoqwa@amazon.com> Co-authored-by: Lucas Jia <lucasjia@amazon.com> Co-authored-by: Ealynn Hsu <89547630+ehsu3@users.noreply.github.com> Co-authored-by: Ealynn Hsu <ealynnh@amazon.com> Co-authored-by: Mahima Chaudhary <36598677+cmahima@users.noreply.github.com> Co-authored-by: Mahima Chaudhary <mahchy@amazon.com> Co-authored-by: Lisa Ni <nilis@amazon.com> Co-authored-by: vaishnavi-kommaraju <vaishnavi.k14@gmail.com> Co-authored-by: Vaishnavi Kommaraju <vvak@amazon.com>

Master nova follow ups (#6051) * Feat: show_metrics() and stream_logs() helper functions (#6002) * Feat: Add show_metrics() and stream_logs() for monitoring training jobs * Feat: show_metrics() and stream_logs() --------- Co-authored-by: Ealynn Hsu <ealynnh@amazon.com> * show_metrics() Enhancement: Display MLFlow metrics for OSS models (#6013) * show_metrics() Enhancement: Display MLFlow metrics for OSS models * Update unit tests * Address code comments --------- Co-authored-by: Ealynn Hsu <ealynnh@amazon.com> * feat(serve): add opt-in model source tag-based resource reuse (#5993) * feat(serve): add opt-in model source tag-based resource reuse Add reuse_resources to ModelBuilder.build/deploy and BedrockModelBuilder.deploy. On a hit, discover an existing resource by the model-source tag and return it instead of creating a duplicate (warn, do not raise). Honored per call. - New sagemaker/serve/model_reuse.py: tag helpers + service-client discovery - Consolidate Nova manifest/checkpoint reading into sagemaker/core/training/utils.py - SageMaker: build() skips Model creation on reuse (sets built_model to the existing Model); deploy() reuses the endpoint after validating env vars/image/ instance type (PrimaryContainer with Containers[0] fallback for Nova) - Reuse gates are skipped for inference-component builds/deploys so IC create/update (via _deploy_for_ic) is never silently intercepted - Bedrock: reuse custom model + active deployment; response includes modelArn - Reuse discovery uses the cached session/bedrock clients - Support raw S3 URI model input via model_metadata BASE_MODEL_NAME - Unit tests + notebook examples * feat(serve): support Nova inference-component deployment and harden reuse Route Nova model-customization deploys through the shared single-inference -component path when a ResourceRequirements inference_config is supplied, so each Nova checkpoint (full-rank or LoRA-merged) is hosted as one inference component referencing the built Model. Nova without an inference_config keeps the direct model-on-variant path. - Broaden _is_nova_model to identify Nova from a package-less source (raw S3 checkpoint or trainer) via base_model_name, in addition to the model package recipe/hub-content name. - Set EnableNetworkIsolation on the IC endpoint config to match the built Model (always True for Nova), fixing CreateInferenceComponent rejection on mismatched network isolation. - Guard model-package-dependent logic (restricted-package path, PEFT/recipe metadata, lineage tracking) so package-less Nova checkpoints deploy cleanly. - Apply accumulated tags (including the model-source reuse tag) to endpoints created on the shared IC path so they remain discoverable. - build(reuse_resources=True) only short-circuits when the backing Model can be resolved; IC endpoints and stale/deleted configs fall through and build a real Model, preventing a None built_model on later IC deploys. - deploy() warns that reuse_resources has no effect for inference-component deployments, which manage their own reuse by component name. - Surface both the manifest.json and output.tar.gz errors when Nova checkpoint URI resolution fails, instead of masking the primary failure. Add unit tests covering the Nova IC path (routing, network isolation, IC spec) and the model-on-variant fallback. * fix(core): resolve Nova checkpoint manifest across all three output layouts Nova training jobs write their checkpoint manifest to different locations depending on the training platform: HyperPod: <output>/<job>/manifest.json Serverless: <output>/<job>/output/output/manifest.json Serverful: <output>/<job>/output/output.tar.gz (manifest inside) resolve_nova_checkpoint_uri previously only tried the serverless manifest path and the serverful tar.gz, so HyperPod jobs (manifest directly under the job directory) failed to resolve. Add build_nova_hyperpod_manifest_s3_uri and try all three layouts in turn, aggregating every failure into the raised error so the real cause is not masked by the last attempt's message. Add unit tests for the HyperPod builder and for resolution from the HyperPod and serverless layouts. * feat(serve): Model-tag reuse, IC-deploy guard, and instance_type fix Simplify reuse discovery by tagging SageMaker Models (not just endpoints) with the model-source identifier, so build(reuse_resources=True) can find and skip recreating an existing Model directly — no IC-state dependency. - Tag non-Nova Models at build time with the model-source tag (matching the Nova path's existing behavior). Both Nova and OSS Models are now discoverable by tag. - Add _find_reusable_model: build(reuse_resources=True) searches Models by source tag, skipping Model creation on a hit. Also discovers the endpoint for deploy() to reuse later. - Simplify _get_model_for_endpoint back to variant-only lookup (returns None for IC endpoints). No longer needs IC-spec resolution since the Model is found directly by tag. - _reused_endpoint_matches_config returns True for IC endpoints (can't read container config from variant; Model was already matched by tag). - deploy() with reuse_resources=True on an IC deploy logs a warning that the flag has no effect (ICs manage reuse by endpoint_name + IC name). - Fix deploy() to set self.instance_type from the caller's explicit value before calling _deploy_model_customization, preventing recipe-resolved defaults from overriding the user's intent. - Add model-source tag assertion to the existing OSS deploy integ test. * feat(train): add dry_run=True to train() (#6027) * feat(train): add dry_run=True to train() Add dry_run parameter to all trainers (SFT, DPO, RLVR, RLAIF). When dry_run=True: - All existing validation runs inline (IAM role, hyperparameters, recipe constraints, infrastructure availability) - Returns None without submitting a job or consuming compute - Raises with clear error message on validation failure Additionally, validate_data_path_exists() is called unconditionally (regardless of dry_run) before job submission to catch non-existent S3 paths or dataset ARNs early. Design follows nova-forge-sdk pattern: validation always runs as part of the normal code path, dry_run short-circuits before the actual TrainingJob.create API call. Changes: - data_utils.py: add validate_data_path_exists() utility (S3 + DataSet ARN) - base_trainer.py: add dry_run to abstract train(), _train_serverful_smtj(), and _train_hyperpod() - sft/dpo/rlvr/rlaif_trainer.py: add dry_run param, pass through to shared methods, short-circuit serverless path - Notebook examples added to SFT, DPO, RLVR, RLAIF notebooks - Unit tests added to existing test files - Integration test added * feat(evaluate): add dry_run=True to evaluate() Add dry_run parameter to BaseEvaluator.evaluate() and all subclasses (BenchMarkEvaluator, CustomScorerEvaluator, LLMAsJudgeEvaluator). When dry_run=True: - All existing validation runs (IAM role, model resolution, recipe, pipeline rendering) - Dataset S3 path / DataSet ARN validated via validate_data_path_exists() - Returns None without submitting a pipeline execution - Raises on validation failure Dataset validation runs unconditionally (not just during dry_run) for CustomScorerEvaluator and LLMAsJudgeEvaluator which accept user datasets. Changes: - base_evaluator.py: add dry_run to evaluate() signature - benchmark_evaluator.py: add dry_run, short-circuit before _start_execution() - custom_scorer_evaluator.py: add dry_run, validate dataset, short-circuit - llm_as_judge_evaluator.py: add dry_run, validate dataset, short-circuit - Notebook examples added to benchmark, custom_scorer, llm_as_judge notebooks * fix(dry_run): support DataSet objects, deduplicate ARN validation, expand coverage - validate_data_path_exists() now accepts Union[str, DataSet]; extracts .arn from DataSet objects for validation - Removed duplicate ARN validation logic; delegates to _validate_dataset_arn_exists() - _validate_dataset_arn_exists() warns on AccessDenied instead of raising (execution role may still have access) - Removed isinstance(..., str) guards in all trainers and evaluators so DataSet objects flow through validation - Added dry_run=True parameter to CPTTrainer.train() - Added dry_run=True parameter to ModelTrainer.train() - Integration test: valid_dataset fixture no longer re-creates on every run - Integration test: added nonexistent_dataset_arn and nonexistent_dataset_obj fixtures - Integration test: added TestDryRunServerful class (serverful compute path) - Unit test: added test_dataset_object_extracts_arn, test_dataset_object_not_found_raises * fix(dry_run): support all AWS partitions in DataSet ARN validation - Update ARN regex to aws(?:-[a-z]+)* to match aws-cn, aws-us-gov, aws-iso, aws-iso-b partitions - Use regex guard in validate_data_path_exists() for consistent matching - Add unit tests for each partition (standard, China, GovCloud, ISO, ISO-B) and invalid partition rejection * [Feat]: Job Notifications for SMTJ (#6042) * [WIP] Job notifications setup * [WIP] Job notif update * Adding tests, dedupe logic, and example * Update ARN example values, clean up SM session definitions, rename func * Update trainers to include 'notifications' param and add arn regex check * Address PR comments, use botocore errors, return notification rule arn --------- Co-authored-by: Ealynn Hsu <ealynnh@amazon.com> * [Docs] Add documentation for show_metrics(), stream_logs(), and job notification setup (#6065) * docs: Add show_metrics, stream_logs, and job notifications documentation * Update example job names * Move monitoring capabilities to model_customization --------- Co-authored-by: Ealynn Hsu <ealynnh@amazon.com> * documentation: add dry-run and resource reuse docs to existing RST pages (#6061) * feat(evaluate): add dry_run and caller IAM permission validation to a… (#6075) * feat(evaluate): add dry_run and caller IAM permission validation to all evaluators - Add dry_run=True parameter to InspectAIEvaluator and MultiTurnRLEvaluator (Benchmark, CustomScorer, LLMAsJudge already had it) - Add verify_evaluation_caller_permissions() to validate the caller's identity has the pipeline-orchestration permissions before submitting - Define EVALUATION_CALLER_ACTIONS constant with the set of 22 IAM actions needed by whoever calls evaluator.evaluate() - Fix bug: LLMAsJudge InspectAI code path was missing the dry_run check - Wire dry_run through _get_aws_execution_context() for all evaluators - Add unit tests covering dry_run behavior across all evaluator types * change: move EVALUATION_CALLER_ACTIONS to iam_policies.py * fix: SMHP RLVR image selection and storm_rbs recipe cleanup (#6079) * [Fix] Remove task-type from RLVR recipe, update RLVR image selection logic * Add check for RLVR/RFT before removing task_type * fix: prefer SMHP image over SMTJ fallback in _train_hyperpod In _train_hyperpod, try get_hyperpod_training_image first (native SMHP image) and only fall back to SMTJ image with SM-TJ->SM-HP tag replacement if the SMHP image is not available. Previously the order was inverted. --------- Co-authored-by: Ealynn Hsu <ealynnh@amazon.com> * fix: stream_logs_smhp extract training job from obj (#6084) Co-authored-by: Syed Jafri <syedjfr@amazon.com> * Fix mlflow (oss models) metrics viz (#6102) * fix: stream_logs_smhp extract training job from obj * fix: render mlflow metrics as png to handle large number of metrics * code cleanup: move io, base64 to top level imports --------- Co-authored-by: Syed Jafri <syedjfr@amazon.com> * fix(dry_run): skip MLflow app creation during dry_run (#6100) When dry_run=True, _resolve_mlflow_resource_arn now lists existing apps but skips creation and waiting. All trainers (SFT, DPO, RLVR, RLAIF, MultiTurnRL) forward dry_run to MLflow resolution. 9 unit tests. * Update error message on ModelBuilder when deploying from S3 checkpoint (#6111) * Update error message on ModelBuilder when deploying from S3 checkpoint * Update ModelBuilder to automatically find image_uri * Update import and methods * Add create notifications helper method (#6113) * fix: stream_logs_smhp extract training job from obj * fix: render mlflow metrics as png to handle large number of metrics * code cleanup: move io, base64 to top level imports * fix: add helper method to create sns topic * fix: renamed IDs for readability in SNS access policy --------- Co-authored-by: Syed Jafri <syedjfr@amazon.com> * fix(serve): BedrockModelBuilder accepts BaseTrainer as model input (#6104) BedrockModelBuilder(model=trainer) now works for SFTTrainer, DPOTrainer, RLVRTrainer, etc. Previously only ModelTrainer and S3 URIs were supported. * fix(serve): fix two P0 resource-reuse bugs (instance-type reuse + Bedrock permission fail-open) (#6109) Two independent reuse defects surfaced during Nova SDK dogfooding, both under reuse_resources=True. Fixed together as they share the reuse code paths. --- 1. instance_type mismatch silently reused --- deploy() trusted the endpoint candidate cached by build() in _reused_endpoint_name without re-validating it. That cache is resolved at build time, before instance_type (a deploy() argument) is known, so a deploy on a different instance type reused the wrong endpoint. Separately, _reused_endpoint_matches_config skipped the instance_type check entirely for Inference Component (IC) endpoints: the IC early-return (no ModelName on the variant) returned True before reaching the check. deploy() now re-validates the cached endpoint against the requested instance_type and falls back to a fresh discovery on a miss or mismatch. _reused_endpoint_matches_config checks instance_type -- which lives on the production variant and is available for every endpoint, including IC ones -- before the IC early-return, so an instance-type mismatch is never silently reused. --- 2. Bedrock reuse_resources=True fails open on missing permission --- Reuse discovery (find_existing_bedrock_model, find_active_bedrock_deployment_for_model, find_existing_sagemaker_endpoint) swallowed every exception and returned None. When the execution role lacked a read permission (e.g. bedrock:ListTagsForResource), a denied discovery call was indistinguishable from "nothing to reuse", so reuse fell through to creating the resource -- which then failed with a confusing "ValidationException: Model with name '...' already exists" because the prior run's resource still held the deterministic name, never surfacing the real cause. A new _reraise_if_access_denied helper re-raises AccessDeniedException as a PermissionError naming the missing IAM action; all other errors still fail open (warn and return None) so transient failures like throttling do not block a deploy. Tests: 86 unit tests pass across test_model_builder.py and test_model_reuse.py, including regression guards for both fixes (IC instance-type mismatch, build-time cache re-validation, access-denied-raises, and fail-open-on-other-error). Co-authored-by: Elise Harvey <harveel@amazon.com> * fix(train): raise on expired credentials in show_metrics log fetch (#6114) * fix(train): raise on expired credentials in show_metrics log fetch When AWS credentials expire, show_metrics() reported "No CloudWatch logs found for job '<name>'. The job may still be starting, or logs may not be available yet", sending users to debug their training job instead of refreshing credentials. _fetch_smtj_logs() and _fetch_smhp_logs() each wrapped their CloudWatch Logs call in a bare `except Exception`, logged a warning, and returned an empty list, so an ExpiredTokenException was indistinguishable from a job that genuinely has no logs yet. The empty list then became the misleading ValueError above. Design follows the existing notifications.py pattern: read the structured error code, re-raise the auth subset as PermissionError with remediation guidance, and let every other code keep its current degrade-to-empty behavior. A genuinely absent log group (ResourceNotFoundException) still returns an empty list, so a just-started job continues to raise the existing "No CloudWatch logs found" ValueError. Changes: - cloudwatch_metrics.py: add _AUTH_ERROR_CODES and _raise_if_auth_error(); apply at describe_log_streams, get_log_events, and filter_log_events; narrow `except Exception` to `except ClientError` - base_trainer.py: document PermissionError on show_metrics() - test_cloudwatch_metrics.py: 20 unit tests covering all 7 auth codes on both platforms, ResourceNotFoundException degradation, mid-pagination failure, and a regression guard for the existing "no logs found" path * change(train): move AUTH_ERROR_CODES to common_utils/constants.py Addresses review feedback: the auth error-code list was duplicated in cloudwatch_metrics.py and its unit test, so the two lists had to be kept in sync manually. Move it to the existing common_utils/constants.py as a module-level frozenset and import it in both places. Test parametrization sorts the frozenset so test IDs stay deterministic. --------- Co-authored-by: sayemkam <sayemkam@amazon.com> * fix: serverful instance type validations + integ tests (#6124) * fix: serverful instance type validations + integ tests * cleanup: add missing newlines to end of files * style(sagemaker-train): Fix missing trailing newlines * fix(train): Log SMHP enum fetch failure at debug, not warning + added unit tests --------- Co-authored-by: Syed Jafri <syedjfr@amazon.com> * fix: update job key used for show_metrics/stream_logs in MTRL (#6128) * Update error message on ModelBuilder when deploying from S3 checkpoint * Update ModelBuilder to automatically find image_uri * Update import and methods * Improving logging level * fix: update job key used for show_metrics/stream_logs in MTRL * Stream logs unified improvement (#6127) * fix(stream_logs): unify log streaming for MTRL, evaluators, and HyperPod Fixes three stream_logs() bugs identified in bug bash testing: 1. MTRL trainer stream_logs() now uses the correct log group (/aws/sagemaker/Job/AgentRFT) and polls status via Job API instead of TrainingJob API — previously hung forever showing nothing. 2. Adds stream_logs() to BaseEvaluator and EvaluationPipelineExecution with support for pipeline, MTRL eval, and HyperPod backends. 3. Patches _stream_logs_smhp() to provide user feedback instead of silently swallowing ResourceNotFoundException and empty events. Introduces LogStreamer utility (poll-once pattern) and stream_log_loop() shared helper to eliminate code duplication across all callers. * test(stream_logs): add evaluator integ tests using existing completed jobs Integration tests for evaluator.stream_logs() against completed pipeline executions in us-west-2/729646638167. Covers BenchMarkEvaluator, CustomScorerEvaluator, and LLMAsJudgeEvaluator. No new jobs launched. * Improve logging, error messages and minor bug fixes (#6135) * Update error message on ModelBuilder when deploying from S3 checkpoint * Update ModelBuilder to automatically find image_uri * Update import and methods * Improving logging level * fix: update job key used for show_metrics/stream_logs in MTRL * fix(serve): Speed up reuse_resources with Tagging API and resolve string training jobs Use resourcegroupstaggingapi.get_resources() for O(1) tag lookups instead of scanning all models/endpoints, and auto-resolve string _latest_training_job to TrainingJob objects in ModelBuilder so trainers work without manual .get(). * Remove region logging * fix(train): Improve trainer UX and reduce verbose logging - Cache sagemaker_session in BaseTrainer.__init__ to avoid creating duplicate sessions on every method call - Remove redundant role validation (was validating 3x per train() call), now validates once in ModelTrainer.__init__ - Demote noisy INFO logs to DEBUG (role validated, stopping condition defaults, recipe paths, output compression) - Add num_lines param to stream_logs() to limit output for long jobs - Prefix CloudWatch log lines with [CloudWatch] and use print() to distinguish container output from SDK logging - Improve show_metrics() error message when time range yields no logs - Raise ValueError on AccessDenied in dry_run data path validation instead of silently warning - Improve model_package_group error message to mention compute option - Move local imports in _train_serverful_smtj to top-level - Remove redundant get_role() call in ModelTrainer.from_recipe() * resolve conflict * Update import * fix(serve): Move endpoint reuse discovery to deploy time (#6142) * fix(serve): Move endpoint reuse discovery to deploy time * cleanup: Remove stray dev logs --------- Co-authored-by: Syed Jafri <syedjfr@amazon.com> * fix: usuability update from feedback (#6150) * fix: usuability update from feedback * update documentation * Doc update: reorganize examples folder (#6155) * fix: usuability update from feedback * update documentation * docs: organize model-customization examples into serverless/serverful/deployment/evaluation subfolders * add job notification integ test * feat: bedrock model reuse for OSS models (#6157) * feat: bedrock model reuse for OSS models * code cleanup: fixed log statements, renamed variables, added type checks --------- Co-authored-by: Syed Jafri <syedjfr@amazon.com> * test: add integ tests for show_metrics and model reuse (reuse_resources=True) (#6158) - Add show_metrics() assertions to existing Nova SFT tests (serverless + serverful) - Add show_metrics() via MLflow to OSS Llama SFT test - Add show_metrics() to MTRL trainer integration test - Add reuse round-trip tests to test_model_customization_deployment.py (OSS, us-west-2) - Add reuse round-trip tests to test_nova_model_customization_deployment.py (Nova, us-east-1) - Add build(reuse_resources=True) tests to both deployment test files * feat(telemetry): Add tracking for dry_run, notifications, reuse (#6159) * fix: usuability update from feedback * update documentation * docs: organize model-customization examples into serverless/serverful/deployment/evaluation subfolders * add job notification integ test * feat(telemetry): Add tracking for dry_run, notifications, reuse_resources, show_metrics, and stream_logs * update telemetry * update telemetry * fix: tail_lines in stream_logs returns last N events (true tail semantics) (#6164) - Add LogStreamer.poll_tail() for fetching last N events: - Stream mode (SMTJ): backward pagination via get_log_events nextBackwardToken - Filter mode (SMHP): filter_log_events with startFromHead=False, paginate until N matches collected (CW bounds pages by scan volume, not result count) - Multi-stream jobs: merge events across streams by timestamp, return globally last N - stream_log_loop: when tail_lines is set, call poll_tail() and return immediately - Refactor _stream_logs_smhp to delegate to LogStreamer + stream_log_loop, eliminating ~80 lines of duplicated inline polling logic - Validate start_time >= 2024-01-01 in _tail_filter_mode (CW API restriction) - Remove dead code: lines_printed counter no longer needed in SMHP forward loop - Add unit tests for poll_tail (stream mode, filter mode, multi-stream merge, multi-page pagination, pre-2024 validation) - Add unit tests for stream_log_loop tail_lines integration * Remove duplicated dry_run return * fix: unit tests * fix: integ test (dryrun, model reuse) * fix: model reuse for Nova model * fix: sagemaker-train test fix, remove extra dryrun tests * fix: 'ModelPackage' is not defined * fix: gpu integ test fixes * fix: key check before logging training job name (#6168) Co-authored-by: Syed Jafri <syedjfr@amazon.com> * Fix tests in trainers (#6169) Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com> * Master nova follow ups (#6170) * Fix tests in trainers * fix(iam): Remove iam:PassRole from caller validation to avoid false denials SimulatePrincipalPolicy without iam:PassedToService context value returns implicitDeny for condition-scoped PassRole policies like AmazonSageMakerFullAccess, causing false test failures. --------- Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com> --------- Co-authored-by: Ealynn Hsu <89547630+ehsu3@users.noreply.github.com> Co-authored-by: Ealynn Hsu <ealynnh@amazon.com> Co-authored-by: LN <133025223+amazeAmazing@users.noreply.github.com> Co-authored-by: Syed Jafri <syedjfr@amazon.com> Co-authored-by: Zhaoqi <jzhaoqwa@amazon.com> Co-authored-by: eliseharvey <108292155+eliseharvey@users.noreply.github.com> Co-authored-by: Elise Harvey <harveel@amazon.com> Co-authored-by: Sayem Kamal <sayemkamal12@gmail.com> Co-authored-by: sayemkam <sayemkam@amazon.com> Co-authored-by: papriwal <papriwal@amazon.com> Co-authored-by: rsareddy0329 <rsareddy0329@gmail.com> Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com>

fix: filter full recipe template from serverless train( ) (#6021) * fix: filter full recipe template from serverless train( ) * fix: update hyperparam ref in mtrl trainer * fix: enable only_user_overrides for RLVRTrainer serverless path * chore: removing extra logging * testing: added new integ test for nemotron * fix: use self.compute instead of user override flag for apply_recipe_hyperparams fix * chore: undo minor indent changes in trainers * code cleanup: recipe override tests --------- Co-authored-by: Syed Jafri <syedjfr@amazon.com>

rlvr validation bugfix (#6000) * fix: removed reward verifier is_lambda + compute check * chore: add log to test to print job arn * fix: use hyp override in nova test as workaround for recipe resolver error * fix: added comments + fixed evaluator test --------- Co-authored-by: Syed Jafri <syedjfr@amazon.com>

wip: nova hyperpod integ tests (#5990) * wip: nova hyperpod integ tests * feat: add polling logic to test * fix: generate bucket name using env var (to pass fortress security scan) --------- Co-authored-by: Syed Jafri <syedjfr@amazon.com>

fix: Made CPT integ tests dry run for optimize for capacity constraints (#6194) * fix: Made CPT integ tests dry run for optimize for capacity constraints * fix: add dryrun to test name for visibility in logs --------- Co-authored-by: Syed Jafri <syedjfr@amazon.com>

fix: rever preset reward function deletion from hyperparams dict (#6181) Co-authored-by: Syed Jafri <syedjfr@amazon.com>

Fix sm-train unit tests + use single logger in base trainer (#6030) * cleanup: use a single logger in base_trainer * fix: update recipe resolver unit tests

datamixing recipe path fix (#6073) * cleanup: use a single logger in base_trainer * fix: update recipe resolver unit tests * Release 3.16.0 (2026-07-15): Bump VERSION files and internal dependency pins to 3.16.0 / 2.16.0 / 1.16.0. Update CHANGELOGs across root and submodules (core, train, serve, mlops). * fix: use absolute path in data mixing recipe path construction --------- Co-authored-by: Syed Jafri <syedjfr@amazon.com>

Add unit test to prevent future regression of preset reward function (#6182) * fix: rever preset reward function deletion from hyperparams dict * testing: add unit test to prevent future regression of preset_reward_function --------- Co-authored-by: Syed Jafri <syedjfr@amazon.com>