Xuanqi He

[Test] Stabilize test_osu by confirming a failing collective before failing the test (#7565) 27% of v3.16.0 CI runs have at least one failing OSU benchmark, concentrated in the 32-node c5n.18xlarge collectives. It is not a performance regression: on 38 fresh 32-instance draws, the first collective in a job intermittently spikes by an order of magnitude at the small packet sizes while the median of that same sweep stays at ~1.0x baseline, and it is gone from the next sweep on. The trip repeated on the immediately following sweep only once in twelve. Measure once, as before. Only when the check trips, measure that benchmark twice more inside a single job on the same nodes and take the verdict on the median per packet size. Per-draw failure rate goes from 11/16 to 1/16, and the one that remains genuinely reproduced. Detection is unaffected: a uniform 1.25x, 1.30x or 1.50x regression injected into real sweeps is still caught 13 of 13 times. Every repetition is recorded in the results table, not just the median. The verdict uses the median, but the spike that triggered the re-measurement only exists in the repetition it happened in, and that is the signal a future investigation would need. A re-measured benchmark therefore contributes three rows instead of one, each in the shape a single run produces, so the historical report is unaffected in shape and the two extra points are the clean ones. The extra time is paid only on runs that trip, so 73% of runs are unchanged and the p95 is 11.9 min, almost all of it alltoall.

Extend MultiNetworkInterfacesInstancesValidator to cover single-card EFA instances (#7457) Since 3.15.0, single-network-card instances with EFA enabled (e.g. hpc6a, c5n) are launched with two network interfaces (a primary interface plus a dedicated efa-only interface), so AWS does not auto-assign them a public IP. Previously MultiNetworkInterfacesInstancesValidator only checked multi-network-card instances, so these queues passed validation and their compute nodes silently failed to bootstrap on public subnets without a NAT gateway. Broaden the validator to also flag single-card EFA compute resources (mirroring the launch-template logic in queues_stack.add_network_interfaces, including the EfaInterfaceType: efa opt-out), and rework the failure messages to name the responsible instance types, explain why they launch multiple interfaces, and give a branch-specific workaround. Co-authored-by: hanwen-cluster <hanwenli@amazon.com>

[Test] Buy capacity blocks when rendering test configs for manually scheduling tests on CB(p6-b200) (#7532) p6 capacity is only realistically obtainable through capacity blocks, which are an upfront, non-refundable purchase, so test_efa on p6-b200 cannot run in the daily integration tests. A `*_CAPACITY_BLOCK_*` variable now resolves like the existing `*_CAPACITY_RESERVATION_*` ones, except it finds or buys a capacity block and renders to its AZ, so the tests run wherever the capacity turned out to be. A block already owned is reused whatever its state and however little is left of it: it is paid for either way, and a replacement would expire at the same time. Otherwise only 'instant' blocks up to a day long are bought, since blocks end at 11:30 UTC and are priced pro-rata, and rendering then waits for the block to become active. Config validation renders every file under configs/ and must not spend money, hence PCLUSTER_SKIP_CAPACITY_BLOCK_PURCHASE, set by the validate-test-configs tox environment.

[Test] Add integ-test for user Slurm maintenance reservations on static and dynamic nodes (#7501) Add test_slurm_maintenance_reservation verifying that a user-created Slurm reservation with flags=maint follows standard Slurm semantics: it keeps unreserved jobs off the reserved nodes but does not change node power state or prevent launching. On a single cluster (1 static + 2 dynamic) the test checks both node types: - static node: an unreserved job pinned to it stays pending, and when the node becomes unhealthy clustermgtd replaces it once and lets the replacement bootstrap back to idle while keeping its reservation (no kill/relaunch loop); - dynamic node: an unreserved job stays pending and the node is not powered up, a `--reservation` job resumes the node and runs to completion; - for both node types, once the reservation is deleted the node is no longer MAINTENANCE+RESERVED and the job that was left pending runs to completion. Also adds `assert_msg_in_log_at_most` to the shared test assertions, as the counterpart to the existing `assert_msg_in_log_at_least`. The static-node section uses both to assert clustermgtd logs the replacement exactly once, which is what distinguishes a single legitimate replacement from a kill/relaunch loop.

[Test] Add integ-test for clustermgtd instance-ID matching under EC2 eventual consistency (#7463) Add `test_clustermgtd_instance_id_matching`, a regression test for the EC2 DescribeInstances eventual-consistency issue where instances are returned with a missing PrivateIpAddress. clustermgtd must keep such instances, match them to Slurm nodes by InstanceId, and leave the healthy nodes in place instead of replacing them by IP matching. The fault is simulated on the head node by setup_missing_private_ip_override.sh, which wraps common.ec2_utils.get_private_ip_address_and_dns_name to raise KeyError('PrivateIpAddress') for a chosen instance, exactly as a real DescribeInstances response with a missing PrivateIpAddress would. The wrapper is toggled via a trigger file and gated so the fault can be turned on/off without reinstalling. Also add an assertion in the scaling stress test to guard against the 'not all EC2 info are available' discard path reappearing at scale.

[Test] Run the scaling stress test verification step from HeadNode to avoid connect IO issue (#7584) Under sbatch, srun runs on the batch host, so every slurmstepd aggregates its stdio connection on one of the cheap compute nodes this test uses, exceeding that node's ENA allowance and timing out a few of them: ``` error: connect io: Connection timed out error: _fork_all_tasks: IO setup failed: Slurmd could not connect IO ``` Varying only the srun host with everything else fixed: t3.medium produced 698 IO failures, the head node none. The step keeps its full width, so only the aggregation point moves — chunking it would have removed the very thing being tested, and raising `TCPTimeout` would have hidden the symptom. `run_remote_command` has no default timeout, so the 720s passed here is the 12 minutes `wait_job_completed` allowed for this step before, keeping the timeout behaviour unchanged.

[Test] Stop running test_efa on p4d, and cover gdrcopy on p6-b200 (#7542) p4d capacity has become unobtainable in practice. On-demand running `test_efa` on p6-b200 covers the same ground. Drop the p4d dimension from `develop.yaml` and `new_os.yaml`. The fabtests gdrcopy cases and `FI_EFA_USE_DEVICE_RDMA` were keyed off p4d alone, so p6-b200 was skipping the GPUDirect RDMA path entirely. Run them on both, p4d included, so the path still works when p4d is run by hand. `released.yaml` is handled by cherry-picking `d6e0d6ea03f9dacc526cf0c39afd8b2d262e2c5c` from `integ-tests-3.15.1`, which had already switched that dimension to c5n.18xlarge for the same reason.

Pin neuron-bench (aws-neuronx-tools) to 2.30 in trainium test (#7494) Note: revert this commit when the bug fixed by aws-neuronx-tools test_trainium's CCL check started failing because output-ccl.txt came back empty while the Slurm job still reported COMPLETED. Root cause: neuron-bench in aws-neuronx-tools 2.31.13.0 (Neuron SDK 2.31.0, released 2026-07-07) segfaults in finalizeRuntimeTracing right after "Benchmark finished", before the CCL summary is printed. The test script installs Neuron with a floating "=2.*", so CI silently moved onto 2.31 as soon as it shipped. Verified with a single-node, ~12s repro (no Slurm/EFA needed): with only the tools version changed and everything else held constant, 2.31.13.0 segfaults (exit 134) while 2.30.10.0 (Neuron SDK 2.30.0, 2026-05-21) runs clean and prints CCL(1/50/99/100). This rules out the runtime-lib / collectives versions, cross-node CC, EFA, Slurm, and the kernel. Changes: - neuron-installation.sh: pin aws-neuronx-tools to 2.30.* - neuron-ccl.sh: add `set -o pipefail` so a neuron-bench crash fails the job instead of producing an empty output-ccl.txt with a COMPLETED state, and so the framework dumps slurm-*.out on failure.

[Test] Double the MPI ring job timeout to absorb cold cluster starts for stabilizing test_efa The timeout handed to mpi_submit_openmpi.sh was 20 seconds on 48-slot instances, which left no headroom over the observed runtime. The first MPI launch in a freshly created cluster reads its binaries and libraries from an EBS root volume whose blocks are still lazy-loading from the AMI snapshot, inflating that one launch by an order of magnitude and making test_efa fail on grounds that have nothing to do with EFA.

Revert "Pin neuron-bench (aws-neuronx-tools) to 2.30 in trainium test (#7494)" (#7555) This reverts commit 0e3abc78cc4cba50b5b0d1e3ee32169bee42925a. Neuron tools 2.31.15 is released which fix the regression bug. See https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/2.31.1.html