DeepSpeed — Engineering Performance
6 engineers all time · Jan 2025 – Nov 2026 · built 2026-11-22 · GitHub
Performance snapshot
Today's rolling 90-day reading for DeepSpeed, compared with the start of the series. Pick a window to move that comparison point.
Avg. perf / dev / mo
+513.3%
0.26 → 1.58 ETV
Active engineers
+25.0%
4.0 → 5.0
Features
+8.3pp
15.9% → 24.2%
vs. Microsoft
0.15x
0.19x → 0.15x · −85% below
DeepSpeed vs. Microsoft
Per-engineer ETV for DeepSpeed against Microsoft as a whole. Both lines are 90-day rolling averages scaled to a 30-day month, so they share one axis and can be read against each other at any point. Pick a window to zoom the chart to it.
Performance Composition
Each month's output split by type of work: Features (new value), Maintenance (sustaining systems), Tests, Docs, and Fixes (rework). The yellow line is output per engineer, so when it rises each engineer is delivering more, whatever the team size did. Unit: Engineering Throughput Value (ETV).
Engineering capacity
Effective engineers behind DeepSpeed, against its pre-AI baseline. Each subject has its own: DeepSpeed's is 0.40 ETV / dev / mo, its first reading in Q1 2025. Per-engineer ETV divided by that gives a capacity multiple, and that multiple applied to the engineers active in the trailing 90 days turns it into engineer-equivalents. The line is the real headcount, so the gap between line and area is what the leverage is worth. Because each baseline is its own, every subject opens at 1.0x on its first day: multiples measure improvement and are not comparable between subjects.
Knowledge concentration
How dependent is this repo on a small number of engineers? Higher top-1 share = higher key-person risk.
Masahiro Tanaka owns 33.0 % of commits.
Behind the numbers
Written summary of the work completed each month.
No monthly reports available yet.
Top engineers
Most impactful commits
Top 10 by ETV in the all-time window.
- 4.0ETVAdd AutoEP + AutoTP parallel folding (#8064) This PR adds **parallel folding** for AutoEP: tensor parallelism (AutoTP) for the dense/attention path can now coexist with expert parallelism (AutoEP) for the routed-expert path on the same set of ranks, **without forcing EP to be a subset of DP** — an EP group may span TP lanes and dense-DP ranks (cross-lane EP). (This PR should be adjusted for ZeRO3 support after #8060 is merged) ## Design Attention/dense and MoE are treated as two independent partitionings of the same rank set, parameterized per parameter family: - Dense / attention / shared-expert params: `stage_size = tp * dp` - Routed-expert params: `stage_size = ep * etp * edp` `dp` and `edp` are always derived, never user-configured, so the invariant `tp * dp == ep * etp * edp == stage_size` cannot be broken from config. The only structural requirement is that the expert width tiles the stage (`stage_size % (ep * etp) == 0`); EP groups are then laid across a TP-lane-major rank ordering, so they may span TP lanes and dense-DP ranks. ### Configuration No new config section. Folding is expressed by the coexistence of the existing `tensor_parallel` and `expert_parallel` sections: ```json { "tensor_parallel": { "autotp_size": 4 }, "expert_parallel": { "enabled": true, "autoep_size": 4, "expert_tensor_parallel_size": 1 } } ``` `expert_tensor_parallel_size` is carried as a config field but currently must be `1` (expert-internal TP is reserved as follow-up and rejected fail-fast). Validation enforces stage divisibility, TP/sequence-parallel exclusivity, and `preset_model` consistency between the two sections. ### Cross-lane expert parallelism Expert parallelism no longer has to be a subset of data parallelism. Shapes where the expert width exceeds (or does not divide) the dense data-parallel size are supported, for example: - `world=4, TP=4, EP=4` (`dp=1`): the EP group is the whole TP group — one expert per rank. - `world=4, TP=2, EP=4` (`dp=2`): the EP group spans both TP lanes and both DP ranks. - `world=8, TP=4, EP=4` (`dp=2, edp=2`): EP groups span TP lanes with expert replication. The per-family gradient convention is keyed to each parameter's replication structure, not to the EP layout, so it holds across the whole `tp*dp` pool: - **Router/gate and dense/LayerNorm** are AVERAGE over the TP (token-replication) group. The folded router runs redundantly on every TP peer; its partitioned work is reconstructed into a replicated full view by `restore_combined`, whose all-gather backward injects a `tp_size` factor that AVERAGE divides out. - **Routed experts** cancel that same `restore_combined` `tp_size` factor (divide by `tp_size`, no TP all-reduce) and reduce data-parallel over the expert-data-parallel (EDP) group. Without the cancellation, folded expert gradients are over-scaled by `tp_size` — invisible to scale-invariant Adam, but real for non-adaptive optimizers and for gradient clipping (it inflates the expert contribution to the global grad norm). This is now fixed for all folded shapes (the MVP TP2×EP4 shape included). ## What's included - Folded process-group derivation using the generalized expert/data-parallel group creation (`mp_mode` TP-strided vs SP-consecutive ordering), including cross-lane EP group tables. - Route-full / partition-dispatch path for folded MoE (`deepspeed/moe/ep_tp_dispatch.py`), with AutoTP skipping AutoEP subtrees. - **Per-family folded gradient reduction**: AVERAGE for replicated router/gate and dense/LayerNorm; a dedicated `tp_size` cancellation for routed experts; SKIP for genuinely TP-sharded params; SUM contracts reserved for a future true sequence-parallel path. - Per-parameter-family ZeRO checkpoint metadata (routed-expert vs dense/router/shared placement) and folded ZeRO-1/2 optimizer-state handling. ## Correctness & validation - Router/gate and LayerNorm gradient parity against a non-folded ZeRO baseline (atol=1e-1, rtol=5e-3, fp32), on TP2×EP4 (8-rank) and the cross-lane shapes TP2×EP4 (4-rank, `ep>dp`) and TP4×EP4 (4-rank, `dp=1`); scale 1.0. - Routed-expert weight parity against a non-folded baseline, verified with SGD (Adam is scale-invariant and would mask a uniform gradient-scale error), for the MVP TP2×EP4 shape and cross-lane TP4×EP4 with `edp=1` and `edp=2`. - New folding unit tests for config, cross-lane group layout, dispatch, runtime, gradient parity, and checkpoint save/load (multi-rank GPU cases gated for GPU runners; CPU/Gloo parity runs on CI). - Real-H100 confirmation (8×H100): router/gate, LayerNorm, and routed-expert gradient parity to the non-folded baseline hold (scale 1.0) for MVP TP2×EP4 and cross-lane TP4×EP4 (`edp=1` and `edp=2`); cross-lane folded training runs with finite loss and finite, non-zero expert/router gradients. - Passes the full unit test suite (`aws-torch-latest-full`) on H100 GPUs. ## Scope / follow-ups - This PR covers AutoEP + AutoTP folding, including cross-lane EP with `etp=1`. The replicated-grad reduction is mode-aware so the sequence-parallel (Ulysses) folding case fits the same contract; AutoTP + AutoEP is the validated path here. - Expert-internal tensor parallelism (`expert_tensor_parallel_size > 1`) is reserved for a follow-up. - ZeRO-3 composition with folding is planned as separate follow-up work (after #8060 is merged). --------- Signed-off-by: Masahiro Tanaka <mtanaka@anyscale.com>Masahiro Tanaka · 104193b4 · 2026-07-12
- 4.0ETVSupport AutoEP with ZeRO-3 zero.Init source modules (#8060) This PR enables ZeRO3 support for AutoEP-managed MoE layers by partitioning expert parameters over expert replica groups while router and replicated parameters use the global data-parallel group. With ZeRO3 enable, AutoEP preserves global data-parallel gradient averaging for AutoEP expert parameters while reducing them over expert replica groups. ZeRO parameters are gathered before AutoEP reads router or expert tensors when replacing MoE modules created under `deepspeed.zero.Init()`. --------- Signed-off-by: Masahiro Tanaka <mtanaka@anyscale.com> Co-authored-by: Ma, Guokai <guokai.ma@gmail.com>Masahiro Tanaka · 02663d6e · 2026-06-26
- 2.8ETVfeat(rollout): segment-based kernel injection for hybrid-engine greedy decode (#8602) ## Summary Adds a **segment-KI framework** that accelerates greedy decode in the hybrid-engine rollout by injecting native CUDA kernels into comm-free segments of the model, plus a **C++ decode loop** that runs the full autoregressive generation with near-zero per-step Python overhead. ### Segment detection: structural, not per-model Detection is attribute-name based: any HF family whose gated MLP uses `gate_proj/up_proj/down_proj` (LLaMA, Qwen, Mistral, DeepSeek, Gemma...) or whose GatedDeltaNet block uses the `in_proj_{qkv,z,b,a}` set is picked up without per-model code. Projections carrying TP collectives are never fused (a defensive `isinstance` check against the Allreduce layer hierarchy). ### Co-location by construction Both GLU (dual-weight GEMV) and GDN (quad-weight GEMV) replacements read the original weight Parameters directly through `autograd.Function`s with exact backward, so training and generation share one forward path: **zero weight copies, no inject/eject switching between rollout and training**. Verified end-to-end by an integration test (below): generate → train one step → generate, output changes, no sync calls. ## Components | File | Role | |---|---| | `csrc/module_inject/fused_glu.cu` | dual/quad-weight GEMV (MLP / GDN input projections), GDN gates, decode attention (warp-per-head, KV split-K, online softmax), fused residual+RMSNorm, triple QKV GEMV, graph-capturable decode step | | `csrc/module_inject/decode_loop.cu` | C++ generation loop taking a backend-agnostic `std::function` replay callable (CUDA graph, eager forward, or future SYCL); argmax and buffer updates run as a private kernel; EOS checked via amortized D2H sync every 16 steps | | `deepspeed/module_inject/segment_ki.py` | segment detection + forward replacement (`apply_segment_ki`) | | `deepspeed/module_inject/kernel_reference.py` | pure-PyTorch executable specifications for every kernel — test oracle, runtime fallback, porting reference | | `deepspeed/ops/module_inject/{fused_glu,decode_loop}.py` | op builders (JIT compile, lazy load) | | `deepspeed/utils/static_cache.py` | hybrid-slot static cache: real static KV layers for full attention, pass-through slot reusing HF's cudagraph-safe GDN state buffers by reference | | `deepspeed/runtime/rollout/hybrid_engine_rollout.py` | prefill → cache setup → graph capture → C++ decode loop, with layered fallback (full-step graph for b=1, graph+argmax for b>1, C++ loop, Python loop) | ## Performance Qwen3.5-4B, RTX 4080 SUPER, greedy, b=1: | | tok/s | relative | |---|---|---| | HF eager | 22.5 | 1.0× | | **DeepSpeed (this PR)** | **69.4** | **3.1×** | | vLLM | 74.9 | — | ## Tests `tests/unit/module_inject/test_segment_ki_kernels.py` — 11 tests, all passing (executed on RTX 4080 SUPER): - [x] **Kernel vs reference consistency** (GPU, 6 tests): every kernel compared against its `kernel_reference.py` oracle on identical inputs (dual_gemv, quad_gemv, gdn_gates, triple_gemv, fused_add_norm, decode_attn) - [x] **Injection invariants** (CPU-runnable, 4 tests): injected forward matches un-injected output, no weight copies installed, composite autograd path, backward scatters gradients to the original Parameters - [x] **Rollout → train → rollout integration** (GPU, Qwen3.5-0.8B-Base): generate with graph capture + segKI, one SGD step through the same injected model, generate again with changed output — no inject/eject/sync calls anywhere in between - [ ] Multi-GPU (AutoTP) validation (planned follow-up) ## Notes - SYCL/XPU porting: `kernel_reference.py` serves as the executable spec; `decode_loop`'s `std::function` interface is backend-agnostic (graph replay or eager forward); SYCL kernels would follow the same op-builder pattern. - Related upstream fix split into a standalone PR: #8666 (attention_unfused alpha squaring). --------- Signed-off-by: Guokai Ma <guokai.ma@intel.com> Signed-off-by: Ma, Guokai <guokai.ma@intel.com> Signed-off-by: Ma, Guokai <guokai.ma@gmail.com>Ma, Guokai · 97aa3c10 · 2026-11-19
- 2.1ETVFix torch-latest workflow unit failures (#8950) ## Summary - Run the workflow Python on 3.12 so transformers main can be installed. The Modal image was already 3.12; the runners and the AWS capacity fallback were still preparing 3.10. - Fix the unit failures those suites hit on current PyTorch and CUDA: stop forking pytest after CUDA is initialized, size the full-suite workers to the GPU count, and correct Gloo subgroups, fp-quantizer zeros, FP8 GEMM shared memory, ZeRO-3 linear autocast, AutoSP dead-code elimination, and CUDA synchronize. - Pin the Accelerate installer, and compare AutoEP fused parity with an fp32 reference so fp16 reassociation does not fail the test. ## Test plan - [x] Full `unit/` suite on 4× H200, torch 2.9.1+cu130, before the review follow-up: parallel 5268 passed, 0 failed; sequential 203 passed, 26 skipped - [x] `cpu-torch-latest` on the merged head, including process-group staging and the Accelerate installer tests - [x] GPU rerun of the quantizer checks added for review: incomplete groups, 6/8/12-bit model output, and zero and subnormal codes. Passed on 4× B200, torch 2.9.1+cu130. - [ ] `aws-torch-latest` and `aws-accelerate` on L40S. Both workflows are disabled, so this head cannot dispatch them. - [ ] `aws-torch-latest-full` dispatched on this head: https://github.com/deepspeedai/DeepSpeed/actions/runs/41483448132. The job timeout is 180 minutes; the H200 suite took about 3h54m plus environment setup. The same full suite passed on 4× B200, torch 2.9.1+cu130: parallel 5439 passed, 226 skipped, 2 xfailed; sequential 203 passed, 26 skipped. - [ ] `modal-torch-latest` GPU job dispatched: https://github.com/deepspeedai/DeepSpeed/actions/runs/41483443764. `modal-accelerate` is disabled. Test collection on this PR passed. `mps-torch-latest` stopped in the runner device check (`no MPS device on this runner`) before any unit test. --------- Signed-off-by: Olatunji Ruwase <tunji.ruwase@snowflake.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Ma, Guokai <guokai.ma@gmail.com>Olatunji Ruwase · a38957c1 · 2026-11-04
- 2.1ETVKeep invalid group-norm sentinels from becoming a finite 1.0 step (#8638) ## Summary - DeepSpeed encodes inf/NaN group norms as `-1`. Combining those with `vector_norm` squares the sentinel to `1.0`, so clipping and Adam treat a failed group as healthy and apply an unclipped step. - Treat non-finite or negative group norms as overflow (`inf`), skip the optimizer step, reset overflow each step, and advance the loss scaler once (including ZeRO-3 / SuperOffload / ZenFlow / FP16 fused+unfused / BF16). - Direct `unscale_and_clip_grads` callers map the sentinel to `inf` so they still zero gradients instead of using clip_coef=1. This is split out of the ZeRO-2 oversized-gradient / stream-lifetime work so the sentinel contract can land independently. ## Test plan - [x] CPU, rebased onto current master (`75eb961d1`, includes #8730): 17 passed in `tests/unit/v1/zero/test_invalid_grad_norm.py` (`combine_grad_norm_groups` / `get_global_norm` / `CheckOverflow`, ZeRO-1 clip, CPU-offload combine, ZeRO-3 / BF16 / SuperOffload overflow paths) - [x] GPU (2× H200, prior run): `TestZeROBFloat16Stage3InvalidNormRecovery`. Skipped on the rebase machine (2 GPUs required, 1 available) - [ ] CI unit tests for the touched runtime files, started by the rebase push --------- Signed-off-by: tunji-ruwase_snow <tunji.ruwase@snowflake.com> Co-authored-by: Cursor <cursoragent@cursor.com>Olatunji Ruwase · 991ccf0f · 2026-10-06
- 1.8ETVSupport custom partitioning patterns for AutoTP (#7806) This PR introduces a flexible, configuration-driven API for AutoTP (Automatic Tensor Parallelism) that allows users to define custom layer partitioning patterns for training. @inkcherry @delock ## Motivation Previously, AutoTP relied on hardcoded layer detection logic that was difficult to customize for new model architectures. This PR enables: 1. **Custom models**: Users can define exact regex patterns to match their model's parameter names 2. **Fused layers**: Support for fused QKV, gate_up_proj, and other packed weight matrices with unequal sub-parameter sizes (e.g., GQA with different Q/K/V dimensions) 3. **Extensibility**: Easy to add new model presets or customize existing ones Here is an example of a config including custom partitioning patterns: ```json { "tensor_parallel": { "autotp_size": 4, "partition_config": { "use_default_specs": false, "layer_specs": [ { "patterns": [".*\\.o_proj\\.weight$", ".*\\.down_proj\\.weight$"], "partition_type": "row" }, { "patterns": [".*\\.[qkv]_proj\\.weight$"], "partition_type": "column" }, { "patterns": [".*\\.gate_up_proj\\.weight$"], "partition_type": "column", "shape": [2, -1], "partition_dim": 0 } ] } } } ``` Refer to the [document](https://github.com/tohtana/DeepSpeed/blob/tohtana/autotp_custom_patterns/docs/code-docs/source/training.rst) for more details (including preset models and how to define partitioning for fused models). We also opened a new [PR](https://github.com/deepspeedai/DeepSpeedExamples/pull/998) to show the usage. ## Simplified initialization step AutoTP previously required calling ``set_autotp_mode(training=True)`` and ``deepspeed.tp_model_init`` before ``deepspeed.initialize``. Now we can include all the necessary configurations in the DeepSpeed config. We still support the traditional initialization path for backward compatibility. When you use both (i.e. calling ``set_autotp_mode(training=True)`` and ``deepspeed.tp_model_init`` and passing the config to ``deepspeed.initialize``), we will merge the settings at initialization. When we have conflicting settings, we will error out. --------- Signed-off-by: Masahiro Tanaka <mtanaka@anyscale.com>Masahiro Tanaka · 6b9cab1d · 2026-01-31
- 1.8ETVSplit DeepCompile ZeRO-3 memory scheduler (#8233) Splits the DeepCompile ZeRO-3 scheduling and gather/release lifetime changes out of #8169. The exact candidate passed 10/10 tests on a single node with 2 H100 GPUs. Signed-off-by: Masahiro Tanaka <tanaka.masahiro@gmail.com>Masahiro Tanaka · 38561463 · 2026-09-10
- 1.7ETV[AutoTP] Fix ZeRO-3 checkpoint consolidation to gather across TP and DP (#8168) AutoTP + ZeRO-3 silently produced incomplete checkpoints: both export paths handled only the ZeRO data-parallel dimension and dropped the tensor-parallel shards. - ds_to_universal.py: stage3 conversion recovers the (tp,dp) grid from checkpoint file names, extracts shards under the real tp_index, and reuses the stage<=2 TP-aware merge when tp_degree>1 (DP-only path preserved for tp_degree==1 -> no regression for plain ZeRO-3). - engine.py: _zero3_consolidated_16bit_state_dict nests GatherReplacedLayerParams inside GatheredParameters so save_16bit_model gathers both DP and TP; remove the blanket autotp+zero3 training block now that checkpoint consolidation is implemented. - stage3.py: load_hp_checkpoint_state resolves the TP shard before the ZeRO-DP partition, so universal checkpoint restore round-trips. Add end-to-end universal conversion tests and update existing tests for the refactored merge_tp_slices / extract_zero_shards_stage3 signatures. --------- Signed-off-by: Guokai Ma <guokai.ma@intel.com>Ma, Guokai · eec237ee · 2026-08-02
- 1.5ETVKeep fp32 masters on ZeRO-1 and ZeRO-2 when a resume skips optimizer state (#8891) ## Summary - `load_optimizer_states=False` used to skip the ZeRO shard, so a fresh optimizer rebuilt the fp32 master weights from the bf16/fp16 parameters and dropped the precision saved in `SINGLE_PARTITION_OF_FP32_GROUPS` or universal `fp32.pt`. - `DeepSpeedEngine.load_checkpoint` no longer accepts `load_fp32_weights`. `zero_optimization.load_from_fp32_weights` is deprecated and ignored. A full ZeRO load always restores the saved fp32 masters. `load_optimizer_states=False` with `load_module_only=False` restores those masters and starts Adam with fresh moments. `load_module_only=True` rebuilds ZeRO-1 and ZeRO-2 masters from the loaded model weights. Universal module-only loads read those model weights from `fp32.pt` first. ZeRO-3 still reads its fp32 partition, because the model checkpoint does not hold the partitioned weights. - A data-parallel size change that requests optimizer state, and a rigid ZeRO-1/2 checkpoint, raise `ZeRORuntimeException`. An elastic ZeRO-1/2 checkpoint can repartition fp32 masters when optimizer state is not loaded. Universal checkpoints keep the masters across a DP size change. ZeRO-3 still rejects a DP size change. - A legacy checkpoint with no `*_optim_states.pt` files, and a universal folder with no `fp32.pt`, raise `ZeRORuntimeException` on any load that needs the masters. ZeRO-1/2 `load_module_only=True` still succeeds without the shard. A `parameter_alignment` mismatch raises on a full load and on a masters-only load. Module-only remains the warm start for that mismatch. - Pytest 8 registers each `DistributedFixture` subclass, so a test skips when the machine has fewer GPUs than the fixture world size. - Further ZeRO-3 and dedicated ZeRO-1 BF16 work is in #8892, stacked on this branch. Follow-ups that do not fail the current unit tests, including ZeRO-3 NVMe `load_module_only=True`, are tracked in #8890. ## Load decision tree `zero_optimization.load_from_fp32_weights` is ignored. When a universal load has `load_optimizer_states=False`, `load_hp_checkpoint_state_from_checkpoint_dir` reads `fp32.pt` and returns. It does not open `optimizer_state.pt`, and it does not fill Adam steps or moments. ```mermaid flowchart TD start["ZeRO checkpoint was found"] --> module{"load_module_only"} module -->|"True, legacy ZeRO-1 or ZeRO-2"| rebuild["Skip the shard. Rebuild masters from the loaded model weights. Moments stay fresh."] module -->|"True, universal or ZeRO-3"| open["Open the shard"] module -->|False| open open --> shards{"Legacy shard files present, or the checkpoint is universal"} shards -->|"Legacy files missing"| raiseMissing["ZeRORuntimeException: saved fp32 weights are required"] shards -->|present| form{"Which checkpoint"} form -->|"Legacy ZeRO-1 or ZeRO-2"| legacy12["Restore the saved fp32 masters. Load Adam moments only when requested."] form -->|"Universal ZeRO-1, ZeRO-2, or BF16"| uniMoments{"load_optimizer_states"} uniMoments -->|yes| uniFull["Read fp32.pt, the moments, and optimizer_state.pt. A missing fp32.pt raises."] uniMoments -->|no| readFp32["Read fp32.pt and return. A missing fp32.pt raises."] readFp32 --> round{"load_module_only"} round -->|no| keep["Copy the masters onto the model weights. The masters keep the saved precision. Moments stay fresh."] round -->|yes| rounded["Copy the masters onto the model weights, then rebuild the masters from those weights. Moments stay fresh."] form -->|"Legacy ZeRO-3"| z3["Copy the saved fp32 partition into the masters and then into fp16. Load moments only when requested."] form -->|"Universal ZeRO-3"| z3Moments{"load_optimizer_states"} z3Moments -->|no| z3Masters["Read fp32.pt. A missing file raises."] z3Moments -->|yes| z3Full["Read fp32.pt, exp_avg.pt, and exp_avg_sq.pt. A missing file raises."] ``` A `parameter_alignment` mismatch raises on every legacy ZeRO-1/2 shard load. Module-only never opens that shard, so it remains the warm start. AI assistance was used to implement and test this change. This does not duplicate an open DeepSpeed PR. ## Test plan - [x] `tests/unit/checkpoint/test_fp32_master_only.py`, `tests/unit/checkpoint/test_zero_optimizer.py`, and `TestStage12ParamAlignment` on 4× NVIDIA H200 (driver 580.159.03), torch 2.9.1+cu130, Python 3.12.12, pytest 8.4.1, commit `894a2b72`: 99 passed. Legacy and universal loads keep the saved masters and drop moments. Module-only rebuilds ZeRO-2 masters from bf16. A missing legacy shard raises `ZeRORuntimeException`, and module-only still loads. A legacy DP change from 2 to 1 raises. A universal DP change keeps the masters. An elastic DP change repartitions the masters and drops moments. A parameter-alignment mismatch raises for a masters-only load and a full load, and the module-only warm start matches the saved parameters. The world-size-4 elastic DP cases and both ZeRO++ loads passed. - [x] `TestUnevenVocabCrossTpResume::test_resume_from_tp2` on the same machine: 2 passed. - [x] The eight elastic DP cases and both ZeRO++ cases skip when one device is visible: `4 required, 1 available`. - [ ] Still open: fp16, MoE, and NVMe offload, including ZeRO-3 `load_module_only=True` with NVMe optimizer offload. See #8890. Validated cells: ZeRO stages 1, 2, and the ZeRO-3 paths in `test_zero_optimizer.py`; same DP and DP 2 to 1; legacy, elastic, and universal checkpoints; bf16; no offload. Open cells: fp16, MoE, NVMe offload. --------- Signed-off-by: Olatunji Ruwase <tunji.ruwase@snowflake.com> Co-authored-by: Cursor <cursoragent@cursor.com>Olatunji Ruwase · 2dd22dc1 · 2026-10-27
- 1.4ETVNon-reentrant activation checkpoint CPU offload (#8282) ## Summary - Continues [@winglian](https://github.com/winglian)'s work from #8181: a `saved_tensors_hooks` context manager that offloads non-reentrant checkpoint hidden-state inputs to a pinned CPU buffer pool on a side stream (`use_reentrant=False`). - The first two commits are **authored and signed off by Wing Lian** (`wing@axolotl.ai`); they are the original #8181 patches, rebased onto current `master`. This follow-up commit addresses review without rewriting those commits. - Review follow-up: restore `GradientCheckpointingLayer.__call__` when no manager is active (HybridEngine train/rollout), skip offloading the last checkpoint input (`keep_last_count=1`), rename `*_size` knobs to `*_bytes` / `*_count`, and add tests for the HF signature contract and keep-last behavior. - Wires the async offload into DeepSpeed native `cpu_checkpointing`: the copy machinery is factored into a reusable `_ActivationOffloadEngine`, which both the HF hooks class and native `CheckpointFunction` / `non_reentrant_checkpoint` share. Also fixes two pre-existing `non_reentrant_checkpoint` + `cpu_checkpointing` bugs (inputs emptied before forward; `saved_data` never restored during recompute). Original upstreaming context: axolotl-ai-cloud/axolotl#3776, requested in #8181. ## Test plan - [x] `pytest tests/unit/runtime/activation_checkpointing/test_offload_activations.py` on H200 — HF `saved_tensors_hooks` path (26 passed) - [x] `pytest tests/unit/runtime/activation_checkpointing/test_activation_checkpointing.py` on H200 — native reentrant + new `cpu_checkpointing` offload tests (30 passed) - [x] `pytest tests/unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py` on H200 — native non-reentrant + new `cpu_checkpointing` offload test (49 passed) - [x] H200 microbenchmark (activation-dominant): async CPU offload matches blocking's 58% peak-memory reduction at ~5.7% step-time overhead (vs ~6.8x for blocking), i.e. 6.4x faster than blocking offload - [ ] CI unit tests for activation checkpointing Made with [Cursor](https://cursor.com) --------- Signed-off-by: Wing Lian <wing@axolotl.ai> Signed-off-by: Olatunji Ruwase <tunji.ruwase@snowflake.com> Co-authored-by: Wing Lian <wing@axolotl.ai> Co-authored-by: Cursor <cursoragent@cursor.com>Olatunji Ruwase · 858e91ee · 2026-08-21