aws-parallelcluster — Engineering Performance

[integ-tests][OSU] Update c5n.18xlarge benchmark baselines using p75 from DynamoDB Replace the old OSU benchmark baselines for c5n.18xlarge with data-driven values computed from DynamoDB (ParallelCluster-PerformanceTest-Metadata table). Methodology: take the 6 most recent runs per OS/MPI/benchmark combination, remove the worst outlier run, then compute p75 from the remaining 5. This approach filters out noisy runs (e.g. rhel8/intelmpi/osu_allreduce had a single run at 6.29x the median) while still producing thresholds that reflect real recent performance. Updated 72 threshold files across 6 OSes (alinux2, alinux2023, rhel8, rhel9, ubuntu2204, ubuntu2404) x 2 MPI variants (intelmpi, openmpi) x 6 benchmarks (osu_allgather, osu_allreduce, osu_alltoall, osu_bcast, osu_latency, osu_bibw). Out of 1500 individual packet-size baselines compared, 886 were tightened (stricter) and 614 were relaxed (more lenient). Files for centos7, rocky8, rocky9, and ubuntu2004 are left unchanged as there is no recent data in DynamoDB for those OSes. Simulated failure rates against the 5 most recent runs: Old thresholds: 16.63% check fail rate, 67/360 benchmark run failures New thresholds: 1.63% check fail rate, 10/360 benchmark run failures

Remove AWS Batch scheduler and Amazon Linux 2 OS support Changes - Delete awsbatch-cli/ package, awsbatch_builder.py, awsbatch_validators.py, Batch API client, and all Batch integration tests - Remove Batch from schemas, constants, cluster model, IAM policy templates, and CI workflows - Remove AL2 from supported OS list and image builder configs - Mechanically replace Os: alinux2 → Os: alinux2023 across all test config YAMLs - Delete Batch-only utility scripts (generate-ami-list.py, update_pcluster_configs.py, upload-script.py, bump-awsbatch-cli-version.sh) — these have no Slurm equivalent use Notes for reviewers - pcluster configure wizard removes the scheduler selection prompt entirely since Slurm is now the only option. - conditions gating on scheduler type are simplified; is_awsbatch helper and related conditional logic removed rather than left as dead code.

[B200] Make test_osu able to run on p6-b200.48xlarge instance type.

Improve export logs user experience (#7549) * Improve export logs user experience * Add unit tests * Ajust wording * Fix test failures * Update CHANGELOG * Use boto3 for thread safe download --------- Co-authored-by: Helena Greebe <hgreebe@amazon> Co-authored-by: hanwen-cluster <hanwenli@amazon.com>

[develop] Upgrade connexion to ~=2.15.1, upgrade Werkzeug to ~=3.1 to address CVE-2024-34069 (#6932) * Upgrade Connexion to ~=2.15.1 (from ~=2.13.0). * Upgrade Flask to ~=3.0 (from >=2.2.5,<2.3). * Upgrade Werkzeug to ~=3.1 (from ~=2.0) to address [CVE-2024-34069](https://nvd.nist.gov/vuln/detail/cve-2024-34069). * Upgrade serverless_wsgi.py to the latest version. * Changes to encoder.py, flask_app.py and etc. to adapt the version bump.

[CLI][GB200] Add ultraserver instance(p6e-gb200) capacity block support (#6928) - Add ultraserver instance support with capacity block validation - Add CapacityBlockHealthStatusValidator for UltraServer instance check. - The validator also collect the capacity blocks that are in valid CAPACITY_BLOCK_INACTIVE_STATES. (for example: state `scheduled`). In this scenario we throw a warning and report the ids and the states. - Add describe_capacity_block_status api in ec2.py - Add logic to collect all ultraserver instance capacity block ids in a dict, key -> ULTRASERVER_INSTANCE_PREFIX, value -> list of capacity block ids - Add ULTRASERVER_INSTANCE_PREFIX_LIST and ULTRASERVER_CAPACITY_BLOCK_ALLOWED_SIZE_DICT constants - Implement capacity block size validation for p6e-gb200 instances (9, 18 nodes) - Add ultraserver-specific template generation and validation logic - Move get_instance_type_and_reservation_type_from_capacity_reservation to ec2.py - Enhance template and configuration handling - Add capacity block support in cluster and queue stack templates - Pass ultrasever capacity block size to HeadNode dna.json. For p6e-gb200, "9", "18" or "9, 18". - Add has_ultraserver_instance to detect if the queue stack compute resources have ultraserver instance - Tests - Add ultraserver capacity block test suites - Add ultraserver cluster and queue stack tests - Fix test mocks to return proper CapacityReservationInfo objects - Add describe_capacity_reservations method to dummy EC2 client - Update existing tests for refactored capacity reservation function

Move node bootstrap files from /tmp to /opt/parallelcluster/tmp Some custom AMIs mount /tmp with noexec, and create-cluster wrote and consumed bootstrap files under /tmp. Move them under /opt/parallelcluster/tmp instead so create-cluster works on such AMIs.

[Test] Add integration tests to validate support for GB200.

[Test] Test to capture race conditions in update workflow. the test verifies that the cluster is not exposed to the following race conditions: that the cluster is not exposed to the following race conditions: 1. A node that completes the bootstrap after the update is not able to execute the update. 2. A node that completes the bootstrap close to the end of the readiness check can cause the readiness check to fail because it does not have enough time to complete the update. 3. A node that faces a transient failure in the execution of the update does not retry the update. 4. A node that faces a transient failure in the detection of the update does not retry the update. 5. A login node that is started before the update but completes the config phase during the update will skip the update. This happens because cfn-hup will see the updated launch template and will think thereafter that it has deployed the updated config, even if it did not.

[Develop] Introduce Global Cleanup IAM Role for ParallelCluster Build-Image (#6912) * feat: Global Cleanup IAM Role for ParallelCluster Build-Image * Introduce ensure_cleanup_role() with bootrapped tagging and idempotent creation / update logic. * 4-step safe update sequence documented, now only after the inline policy succeeds, set or bump the bootrapped tag. * Lambda VPCAccess managed policy is attached only when LambdaFunctionsVpcConfig exists in the config * Modify image_operations_controller to invoke ensure_cleanup_role when Build/Iam/CleanupLambdaRole is not provided and to fail fast on permission errors. * Refactor imagebuilder_stack to remove all per-stack cleanup-role logic and wire Lambda to the global role by default. * Update constants (role prefix / expected revision tag key & value). * IamClient – add create_role, attach_role_policy, put_role_policy, tag_role