hanwenli

Move node bootstrap files from /tmp to /opt/parallelcluster/tmp Some custom AMIs mount /tmp with noexec, and create-cluster wrote and consumed bootstrap files under /tmp. Move them under /opt/parallelcluster/tmp instead so create-cluster works on such AMIs.

[performance tests] Loosen OSU benchmark baselines Loosening rule, applied per (mpi, benchmark, packet size): new = clamp(that OS's recent median, all_os_median*0.85, all_os_median*1.15) Verified that known real slowdowns still fail (e.g. rhel8 openmpi osu_alltoall at 128/256 B, where measured latency is ~2x the fleet, and ubuntu2404 openmpi osu_alltoall at 1 B). Baselines were derived from the last ~5 months of historical results in the ParallelCluster-PerformanceTest-Metadata DynamoDB table. Replaying that history against the new thresholds reduces failing benchmark-runs from 73 to 53 (~27%) This commit also deletes baselines for CentOS7, Ubuntu2004, Amazon Linux 2

[export-log] Fix false "No log events in the log group" error on log export Prior to this commit, validation error of "no log events" is triggered whenever the first FilterLogEvents call came back with an empty events list. FilterLogEvents is paginated, so a page can be empty even when if more data is available in the next pages. This commit triggers the validation error only if both events list and `nextTokens` are empty

[integ-tests-framework] Make capacity reservations for all instance types 1. With https://github.com/aws/aws-parallelcluster/pull/7440, we started to make capacity reservations for {"c5.xlarge", "m6g.xlarge", "m6i.xlarge"}, and use other similar instance types if a capacity reservation fails to creat. This commit expands the logic to all instance types. 1.1. With instance types <= .xlarge, we make duplicate capacity reservations because multiple tests in parallel could use the same instance types, therefore need multiple capacity reservations. With instance types >.xlarge, we make only one capacity reservation because tests with larger instance types usually make capacity reservations early in the test definition (e.g. test_efa in commercial makes capacity reservation in `develop.yaml`), therefore this second layer of capacity reservation shouldn't make duplicate capacity reservations. 1.2. With instance types supporting EFA, create the capacity reservation in a placement group. With instance types not supporting EFA, create the capacity reservation without a placement group. 2. With this commit, resolve_instance_with_capacity allows specifying alternative_instance_types. Prior to this commit alternative_instance_types was always calculated with `get_similar_instance_types`, which could be too restrictive, so don't give too many alternatives for instance types like `c5n.18xlarge` 3. Improve test_efa in isolated_regions to take a flag to use any efa instances to avoid Insufficient Capacity Error. test_efa in commercial doesn't need this, because it could try out different regions. In isolated regions, the test has to run in a specific region.

Specify working directory of imagebuilder to avoid executing in /tmp This commit also set `noexec` on `/tmp` for some tests to monitor ParallelCluster compliance of `noexec` on `/tmp`: 1. test_essential_features: now `noexec` is set on `/tmp` by default. This commit fixes the `gpu_job.sh` to not execute on `/tmp` 2. test_multiple_nics: now `noexec` is set on `/tmp` by default. This test is added because multi-nics configuration script was executed in `/tmp` before pcluster 3.16.0 3. test_build_image: If the test is run on a non-GPU instance, where Nvidia is not installed, create a custom AMI with `noexec` persisted on `/tmp` and run `build-image` on top of this custom AMI. Using the custom AMI has better coverage than (1) and (2), because the `noexec` persisted on `/tmp` in the custom AMI covers the whole workflow of `build-image` and `create-cluster`.

[integ-tests framework] Honor --ami-owner option when retrieving AMIs in integration tests The --ami-owner option was ignored by `retrieve_latest_ami`, so tests always searched the default per-OS owners (amazon/self). This prevented running `retrieve_latest_ami` against ParallelCluster AMIs built in a separate account.

Fix flaky failures during creating CFN stack from proxy.yaml The proxy.yaml CFN stack creation sporadically failed with "WaitCondition timed out. Received 0 conditions when expecting 1" on ProxyVerificationWaitCondition. Root cause is a startup race. `ProxyClient` runs in the private subnet and reaches the internet only through the Proxy instance, whose transparent-proxy plumbing is set up by its UserData. `ProxyClient` is created after `Proxy`, which CloudFormation satisfies as soon as the `Proxy` instance is running, not when its UserData finishes. When the `Proxy`'s own "apt-get update" is slow (e.g. a slow security.ubuntu.com mirror), `ProxyClient` would fail connecting to network. Console logs confirmed the race: in a failing run the Proxy did not add the iptables REDIRECT rules until 04:51:31, while the ProxyClient's apt window ran 04:50:09-04:50:42 and failed ~49s too early. On runs where the mirror is fast, the Proxy finishes first and verification passes. Fix: add ProxyReadyWaitConditionHandle/ProxyReadyWaitCondition. The Proxy signals this handle as the final step of its UserData. This removes the race regardless of how long the Proxy's apt takes.

[integ-tests] Add post_cluster_setup to run a function after cluster creation This is useful for test_ad_integration with certificate verification enabled where we need to place the directory certificate and restart SSSD before running pcluster-diag

[integ-tests] Verify NFS v3 client mount has lockd pinned to port 4045 Among our integration tests, only FSx Ontap supports v3 mount. Therefore, the check is added to the end of test_multiple_fsx. The check mounts FSx Ontap again with v3, and checks lockd register port 4045 in rpcbind. This commit also allows 4045 port in the security group of the FSx Ontap, because v3 mount also requires lockd port on the server, which is defined as 4045 in Ontap too

Fix ami-patching.yaml cleanup failure 1. The ARN of EBS snapshot should't contain account id. 2. The tagging of EBS snapshots inside the Lambda function is not necessary because imagebuilder propagates the tags to snapshots