github.com-microsoft-FluidFramework

Switch to React's new JSX transform (#26631)

improvement(build-tools): add explicit return types and handle undefined from findConfigFile (#26186) ## Summary - Add explicit return type annotations to functions and methods across build-tools and bundle-size-tools packages to satisfy the `@typescript-eslint/explicit-function-return-type` rule - Handle the `undefined` case from `TscUtils.findConfigFile()` in repo policy check files, fixing potential runtime errors when config files cannot be found - Fix JSZip import in bundle-size-tools to use namespace import (`import * as JSZip`) instead of named import for compatibility ## Changes **Explicit Return Types** Added return type annotations to ~80 functions/methods in: - `gitRepo.ts` - Git command wrappers - `monoRepo.ts` - Monorepo utilities - `npmPackage.ts` - Package accessors - `timer.ts`, `logging.ts`, `utils.ts` - Common utilities - `buildGraph.ts`, `fluidRepo.ts`, `fluidRepoBuild.ts` - Build system - `leafTask.ts`, `tscTask.ts`, `webpackTask.ts` - Task implementations - `PrCommentsUtils.ts`, `getCommentForBundleDiff.ts` - ADO integration - Various other task and utility files **Undefined Handling** - `fluidBuildDatabase.ts` - Added undefined check for `findConfigFile` result - `fluidBuildTasks.ts` - Added undefined checks in two locations for `findConfigFile` result **API Changes** - `unzipStream` return type updated to use `ReturnType<typeof JSZip.loadAsync>` due to import change --------- Co-authored-by: Joshua Smithrud <54606601+Josmithr@users.noreply.github.com>

test(api-markdown-documenter): Add "deep" config end-to-end test cases (#23586) Adds a test case that leverages new hierarchy options to create a "deep" suite configuration where documents that generate folder hierarchy yield documents named "index" _inside_ their folder,

Introduce Unified ServiceClient API for Fluid Client (#27693) ## Description This add a simple API as a cleaner alternative to fluid-static and aquaduct. The implementation details are well encapsulated, making this a viable alternative to the `fluid-static` public API and a viable successor to the existing much larger legacy API surface we do not want to stabilize. This addresses several issues with our existing APIs: 1. non-legacy (fluid-static declarative model) API: 1. No stable way to use local service for testing (Internally or for customers). Forces tests to use alternative patterns (like test-utils, mocks etc. or a separate azure local service process). 2. No way to avoid bundling in SharedDirectory. 3. Defining and using DataStores is not well supported, especially for the root. 4. Dependency layering is such that its disallowed for DDS packages to use the public API surfaces to do collab tests. 5. No abstraction for Services / ServiceClient makes logic portable over services harder to write. 6. Doesn't support lazy loading DDS and DataStore code (at least not clearly/cleanly). 7. No migration path from legacy APIs to this 3. legacy (encapsulated model, including aquaduct): 1. Leak way too many details with too many non-sealed interfaces presenting tons of theoretical extension points we don't want to stabilize. 2. Encourages use of DataObject and its subclassing based extension approach which is limiting (Like how it makes switching to root trees) hard, and makes changing the implementation without introducing breaking changes very difficult. Because this new "unified" API offers value for our own internal testing (for example in tree), it is useful, even as alpha (or internal). Exposing this new API as alpha should be low risk, as it makes no long-term commitments. Since we have first party in-repo use cases, it should be practical to refine and iterate on this API with use in test and examples before we consider promoting it to beta. Requirements for this new API: 1. Testing: 2. Can easily be used for tests including collab and reopening documents. 3. Can be used by DDS (like shared-tree) tests replacing the need for many of our mocks and test utils 4. Can be used by customers (apps and libraries requiring stable APIs when promoted past alpha). When adopted, customers should not have need of any of our legacy test-utils and mocks (so we can migrate our legacy apps off of our test utils and eventually make them internal only). 5. Will be possible to stabilize to public (or at least beta) without depending on or stabilizing legacy APIs. 6. Is easy to use for tests, including collab. 7. Is easy to use for real production apps (should work the same as with tests). 8. Provides a clear and practical incremental migration path from both existing APIs: there can be gaps we need to close for some use-cases but those gaps need to be practical to close.

(pipelines): Add build performance observability pipeline (#26316) ## Description This PR updates the placeholder pipeline added in https://github.com/microsoft/FluidFramework/pull/26299. The new pipeline does the following: - Collects build metrics from ADO REST APIs for PR and internal builds - Generates an HTML dashboard (published as a pipeline artifact) which includes: - Summary metrics (total builds, avg duration, trend analysis) - Duration trend charts over time - Stage and task duration breakdown charts - Tables of recent and longest builds, including links to the source commits/PRs - Will run on a daily schedule Note: The pipeline must run in the public and internal projects to fetch PR build data and internal build data, respectively. ## Next Steps/Other Considerations - Seek feedback from the team on what other metrics could be useful. - It seems ADO only retains a limited number of PR builds. If possible, we should try to increase the limit. - Consider adding an external data store (especially if we cannot increase the public build limit). ## Misc [Spec](https://microsoft.sharepoint-df.com/:fl:/s/8b2be717-e115-43f1-9d10-bd09d33fd20b/IQAQCryKGC8vSo3_JIdYz-tnAW-5TjJglskciG2A7EQ3jN0?e=A68U9w&nav=cz0lMkZzaXRlcyUyRjhiMmJlNzE3LWUxMTUtNDNmMS05ZDEwLWJkMDlkMzNmZDIwYiZkPWIlMjF2TlJQUTAwMUprQ3BuVmxyYnpzMWoxeE9JU0NSOVFGT2dMOHk2RlpXTkk3YTZqZTc2bksxUWFoTExXaGpYQ3hzJmY9MDFTTE9WT1RRUUJLNklVR0JQRjVGSTM3WkVRNU1NNzIzSCZjPSUyRiZhPUxvb3BBcHAmcD0lNDBmbHVpZHglMkZsb29wLXBhZ2UtY29udGFpbmVy) [AB#55451](https://dev.azure.com/fluidframework/235294da-091d-4c29-84fc-cdfc3d90890b/_workitems/edit/55451) ## Example Screenshot 

build(client): generate eslint 9 configs for all packages (#25989) This change updates the shared eslint 9 config to be TypeScript and updates the script in scripts/generate-flat-eslint-configs.ts to support a `--typescript` flag to output typescript per-package configs instead of ESM. Also adds a `--finalize` flag that will do some additional cleanup once packages are fully migrated to eslint 9. Finally, there are lots of fixes to the script to handle ignore entries, inheritance from other configs, etc. The script now produces configs that work in my testing without any modifications. Even if that doesn't hold true once we are truly migrating (see #25932), merging this beforehand will simplify the review of those changes. All the configs were generated by the script using this command: ``` tsx scripts/generate-flat-eslint-configs.ts --typescript ```

feat(tree): staged allowed types (#25116) ## Description Adds `staged` API to `SchemaFactoryAlpha`. This creates an allowed type in the view schema that may or may not be included int he stored schema. Continuation of https://github.com/microsoft/FluidFramework/pull/24631, with a new PR from my fork so I can have push permissions. Major changes: - Adds SchemaFactoryAlpha.staged - There is no longer a single choice for how to derive a stored schema from a view schema: - Different use cases have been split into different APIs for example (toInitalSchema, toUpgradeSchema) - The underlying toStoredSchema now takes options to control how each staged schema is handled. - Unhydrated content always permits staged content (with the exception of clone): this ensures that export/import round tripping of staged content works. - testDocuments test suite has been expanded so existing round trip tests cover the above. Known issues/limitations: - Recursive types are not supported. Tracked by https://dev.azure.com/fluidframework/internal/_workitems/edit/45711 - Clone should produce nodes with a union of source context and staged types so that both unknown optional fields work, and new staged types can be inserted. Tracked by https://dev.azure.com/fluidframework/internal/_workitems/edit/45725 but also partially hidden by https://dev.azure.com/fluidframework/internal/_workitems/edit/45723 which covers how inserting the out of schema types does not error in unhydrated context currently. This is ok, as we catch them when inserting into hydrated documents and thus the two pugs combine to make the desired behavior, but itn't great, and could cause other issue due to violating internal invariants. Some of the new clone tests cover this. - Some places, mainly those not using alpha typer, can't accept annotated allowed types and thus don't accept staged types. Examples include the root of the tree view config, recursiveObject fields, and likely more. Many of these can be worked around using dedicated alpha APIs and/or wrapping the implicit field schema in an explicit one using SchemaFactoryAlpha.required. Some cases, like recursiveArray do not have a viable workaround, and can be addressed in future work, possibly after stabilizing annotated allowed types.

Add tree-agent package (#25114) This new package hosts the SharedTree semantic editing library. Much of the package config has been copied from other FF packages for consistency (e.g. from dds/tree). The library itself is in a usable state but will be iterated upon heavily in the coming weeks. --------- Co-authored-by: Taylor Williams <60717813+taylorsw04@users.noreply.github.com>

Automatic invalidation based on observation tracker for readers of SharedTree content (#25459) ## Description Adds automatic observation tracking for shared tree content. Adds react hooks to use this, along with strong types for use in those hooks to allow safe passing of nodes in props.

(tree): Incremental summarization of chunked forest (#24949) ## Description [AB#1263](https://dev.azure.com/fluidframework/235294da-091d-4c29-84fc-cdfc3d90890b/_workitems/edit/1263) has an overall description and link to a design document. This PR adds incremental summarization support to chunked forest. Currently, the chunked forest data is encoded and added to a single blob in the summary tree. This change breaks it down into mutiple subtrees where each subtree can be incrementally summarized, i.e., if such a subtree doesn't change between summaries, a summary handle will be added to the summary instead of the entire subtree. The schema will support marking properties as incrementally summarizable (TODO - not in this PR). Any chunks in the fields corresponding to this property will be the boundary at which incrementally summarization will work. Basically, if this property changes or anything within it changes (for objects and arrays), its chunks will be re-summarized, or else, a summary handle will be used. To achieve this, this PR makes the following changes: - Added a new encoded format called `EncodedIncrementalChunkShape` to `EncodedChunkShape` for chunks whose data will be incremental summarized. - Added a new shape called `IncrementalChunkShape` which represents chunks whose data can be incrementally summarized, aka, incremental chunks. - Each incremental chunk is assigned a `ChunkReferenceId` which uniquely identifies it under its parent field. - Added a new field encoder called `incrementalFieldEncoder` which encodes its chunks as `IncrementalChunkShape`. The `ChunkReferenceId`s of all chunks in a field are stored as `EncodedNestedArrayShape` in the encoded data. - Added a new decoder called `IncrementalChunkDecoder` which can decode an incremental chunk. - Added `IncrementalEncoder` and `IncrementalDecoder` interfaces which has properties that help with encoding and decoding of incremental chunks. These are added to the `FieldBatchEncodingContext`. - Added a `ForestIncrementalSummaryBuilder` which tracks chunks across summaries and generates summary tree or summary handles for them depending on whether they change between summaries. - The forest summary tree consists of a blob called `ForestTree` at the root that contains the data for the forest. If there are any incremental fields in the forest, an array of chunk reference ids will be encoded for its chunks in the data. - Fore each incremental chunk, a separate summary tree will be added with its reference id as the key for the tree. - The data for the chunk will be added to a blob called `contents` under the tree. If the chunk has any incremental fields, an array of chunk reference ids will be encoded for its chunks in the data. - The same format is repeated until all incremental fields have been summarized. - Note that currently, an incremental field will have exactly one chunk. This may change in the future where the data may be encoded into multiple chunks. [AB#14308](https://dev.azure.com/fluidframework/235294da-091d-4c29-84fc-cdfc3d90890b/_workitems/edit/14308) [AB#41861](https://dev.azure.com/fluidframework/235294da-091d-4c29-84fc-cdfc3d90890b/_workitems/edit/41861) [AB#41863](https://dev.azure.com/fluidframework/235294da-091d-4c29-84fc-cdfc3d90890b/_workitems/edit/41863) [AB#41864](https://dev.azure.com/fluidframework/235294da-091d-4c29-84fc-cdfc3d90890b/_workitems/edit/41864)

refactor(examples): Add explicit function return types (#26074) Part of a multi-PR effort in preparation for enabling global eslint enforcement.

Factor out tree identifier compression and healing utils (#27367) ## Description Factor out tree identifier compression and healing utils. Remove the originator id (which is incorrect) that is passed down into the field batch codec, factoring the encode and decode codecs into different types which only take in exactly what they should have. This intentionally changes `IIdCompressor` from `@public` to `@public @sealed`. While in some ways this is a breaking change, custom implementations of `IIdCompressor` didn't actually work due to us already having code which assumes they are our implementation and down casts them. Therefor adding `@sealed` is more of a bug fix than a break.

refactor(api-markdown-documenter): Update Documentation Domain to be more restrictive (#24667) ### Background The Documentation Domain (`DocumentationNode` and its implementations) was created with a goal of being very flexible in the kinds of trees of content it could express. This was accomplished by, in most cases, allowing parent nodes to contain any kind of child content (with a couple of exceptions). This ultimately ended up creating problems in terms of Markdown-compatibility in particular (HTML is more expressive, so compat there was less of an issue). There are also some forms of hierarchy allowed by the system (and inadvertently used by the system) that don't make sense (e.g., parenting a `SectionNode` under a `ParagraphNode`. As this library has evolved, I have become more and more convinced that the Documentation Domain is unnecessary and should be removed. In this world, `ApiItem` transformations would output `mdast` (Markdown Syntax Trees) directly, instead of going through an intermediary domain. `mdast` has a vast support ecosystem, so it seems like a much better target than some bespoke intermediary domain. ### This PR This PR is an attempt to begin aligning the Documentation Domain with Markdown syntax requirements by restricting the kinds of contents that can appear under specific kinds of nodes. It also introduces a new pattern for domain extension, which itself aligns with how the broader `unist` ecosystem accomplishes extensibility. We require this extensibility internally, as our `fluidframework.com` website build actually does extend the Documentation Domain with a custom node type. In the future, this can be trivially mapped to a custom `mdast` node, but for now we need to support it. #### Immediate impact See the updated HTML test assets for the immediate impact on the output format. Note the reduction in unnecessary hierarchy.

feat(tree): Add Record node kind (#24908) Adds a new Record node type (constructed via `SchemaFactoryAlpha.record`) and updates various alpha-level domains to use it. - TableSchema now uses a Record in place of a Map for its `cells` (this was always the intended design) - JSonDomainSchema now uses a recursive Record in place of a recursive Map for its Object representation [AB#39284](https://dev.azure.com/fluidframework/235294da-091d-4c29-84fc-cdfc3d90890b/_workitems/edit/39284) --------- Co-authored-by: Noah Encke <78610362+noencke@users.noreply.github.com> Co-authored-by: Craig Macomber (Microsoft) <42876482+CraigMacomber@users.noreply.github.com> Co-authored-by: jzaffiro <110866475+jzaffiro@users.noreply.github.com>

feat(client): [internal] container extensions and `presence` use (#24399) ## ContainerRuntime path-based Signal addressing For Signals, support using a `/` prefixed address in `IEnvelope` (at ContainerRuntime). No change to Ops. - Only extension Signals are sent with `/` prefix. - Receiving is supported for `/ext/` (as well as `/runtime/` and `/channels/` for uniformity). Refactor signal submission so that application of `applyTrackingToBroadcastSignalEnvelope` is centralized under `sequenceAndSubmitSignal` and use `UnsequencedSignalEnvelope` to make `clientBroadcastSignalSequenceNumber` stamping (or lack of) more obvious. `ContainerRuntime.submitSignalFn` confirms envelopes coming through don't start with `/` that is reserved for path-based addressing. ## Signal message types promoted to generics `ISignalEnvelope` and `I*SignalMessage*` related types accept `type` and `content` pairs to form tight association between them. ## [internal] `ContainerExtension` API Refactor the prototype API from `presence` package to `container-runtime-definitions`. Extensions are registered with `ContainerRuntime` (via support from `FluidContainer` in `fluid-static` in declarative API) and currently may only send and receive Signals. Encapsulated support will require further work to define appropriate cross-layer interfaces. `fluid-static` support will likely change to register `presence` directly but does not currently. Build out extension message types to define outbound and inbound variants with inbound further divided into raw (unverified) and verified versions. Extensions may only submit messages conforming to the types they specify. The `ExtensionHost` (aka `ContainerRuntime` will deliver typed messages via `processSignal` but in unverified form. ## `core-interfaces` additions - `TypedMessage` added for Signal message generics constraint type. - [internal] `BrandedType` added to support distinguishing raw from verified messages. - [internal] `InternalUtilityTypes.FlattenIntersection` exposed and fixed up for extension message types. ## `presence` changes - Adapt internal `IEphemeralRuntime` to mostly match `ExtensionHost`. - Separate message types to `protocol.ts` and use more broadly. - Update test types (and some data) to conform to the stricter requirements - Improve incoming signal validation: - Check for exactly known type names - Allow for "optional" unrecognized signal; if a signal is not optional it must be recognized to avoid assert (throw) ## Support `getPresence(container: IFluidContainer)` - Update `Container` and `ContainerRuntime` to support extensions. - Replace `getPresenceViaDataObject` use in examples and tests --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update how shared tree codec versions are generated (#25752) This change updates how the write version for the different codecs in tree are determined. Currently, there is a mapping of `SharedTreeFormatVersion` to a list of write versions for all codecs. The `SharedTreeFormatVersion` is one of the options that can be passed when creating a tree and the codecs version are selected from the mapping based on this value. With this change, `SharedTreeFormatVersion` will be removed. Instead, the write versions for the codecs will be selected based on the `minVersionForCollab` option that is also passed during tree creation. The `minVersionForCollab` will be passed down to each codec and it can decide what write version it wants based on this. This PR also standardizes how the various versions are maintained by the codecs (using an `as const` enum-like object and branded values, and a function that generates the right version based on the `minVersionForCollab`). The mapping from `SharedTreeFormatVersion` to `minVersionForCollab` is as follows: * `SharedTreeFormatVersion.v1`: no longer an option * `SharedTreeFormatVersion.v2`: no longer an option * `SharedTreeFormatVersion.v3`: `minVersionForCollab: FluidClientVersion.v2_0` * `SharedTreeFormatVersion.v4`: was never an option, still not an option * `SharedTreeFormatVersion.v5`: `minVersionForCollab: FluidClientVersion.v2_43` * `SharedTreeFormatVersion.vSharedBranches`: no longer an option. Use `SharedTreeOptions.enableSharedBranches` instead. Unchanged: specifying `minVersionForCollab: FluidClientVersion.v2_52` will enable `DetachedFieldIndexFormatVersion.v2`. [AB#51463](https://dev.azure.com/fluidframework/235294da-091d-4c29-84fc-cdfc3d90890b/_workitems/edit/51463). ## Breaking Changes `SharedTreeFormatOptions` (internal) no longer has a `formatVersion` field. --------- Co-authored-by: yann-achard-MS <97201204+yann-achard-MS@users.noreply.github.com>

docs: Port remaining github wiki documentation to `docs` directory (#27731) Also enables a couple more linter rules to enforce common structure (single top-level heading as document title). Contents from the wiki were copied as-is, save for the following changes: - Links were updated to use relative file paths where appropriate - Line formatting was updated to follow our guidelines - All documents were given top-level headings - Other misc. formatting/linting violations were fixed per the linter configurations - An obsolete document on our breaking changes policy (pre-2.0) was removed, and redirect notice pages were removed This PR also sets up LFS for binary asset storage, with associated documentation. --------- Co-authored-by: Alex Villarreal <716334+alexvy86@users.noreply.github.com>

refactor(container-runtime): Enable `no-explicit-any` eslint rule and fix violations (#23570) Enables `any`-related rules from the recommended eslint config and fixes violations. Specifically, - `@typescript-eslint/no-explicit-any` - `@typescript-eslint/no-unsafe-assignment` - `@typescript-eslint/no-unsafe-call` - `@typescript-eslint/no-unsafe-member-access` - `@typescript-eslint/no-unsafe-return` This PR is one piece of a multi-PR process to migrate the `container-runtime` package to our recommended eslint config. [AB#3027](https://dev.azure.com/fluidframework/235294da-091d-4c29-84fc-cdfc3d90890b/_workitems/edit/3027)

refactor(container-runtime): Enable `@typescript-eslint/explicit-module-boundary-types` eslint rule and fix violations (#23597) One in a series of PRs working towards migrating the package to our `recommended` config. Plus a couple of misc. comment syntax fixes/improvements. [AB#3027](https://dev.azure.com/fluidframework/235294da-091d-4c29-84fc-cdfc3d90890b/_workitems/edit/3027)

fix(container-runtime): block local submits during stashed-op apply (#27297) ## Summary - `PendingStateManager` exposes a one-way apply lifecycle (`applying` → `ended`, default `ended`). The window opens eagerly in the constructor when stashed state is present and closes the first time `applyStashedOpsAt` drains `initialMessages` successfully. An apply error leaves the lifecycle at `applying`. - `ContainerRuntime.isReadOnly()` aggregates `pendingStateManager.isApplyingStashedOps` and fans out via `notifyReadOnlyState`, so compliant DDSes skip submits during stashed-op replay. - `ContainerRuntime.submit()` always emits a `SubmitDuringStashedOpApply` error event on a bypass during the apply window. By default the runtime logs and continues; setting `Fluid.ContainerRuntime.EnableSubmitDuringStashedApplyThrow` opts in to the fatal `UsageError` + container close. Only `BlobAttach` is allowlisted (it is legitimate runtime-internal traffic during apply — `sharePendingBlobs` runs before the apply window closes); `IdAllocation` is *not* in the allowlist because assert `0x9a5` enforces that it is submitted directly to the outbox and never reaches `submit()`. Checking at submit time (rather than at `onFlushBatch`) closes the gap where a deferred flush would land after the apply window closes and mask the misuse. - `ContainerRuntime.replayPendingStates` asserts that the apply window is not open. The only real caller is `setConnectionStateCore`'s `canSendOps` edge, and the loader awaits `applyStashedOpsAt` before transitioning to a write-capable connection — the assert surfaces a contract violation rather than silently swallowing the edge. ## Background When a host calls `loadContainer` with `pendingLocalState`, the runtime replays stashed ops via `applyStashedOpsAt`. If a DDS submits a local op during that replay — typically a realize-time write that doesn't consult `readOnly` — the new op has no counterpart in the saved-op stream and the next saved-op replay throws "pending local message content mismatch". This change makes the runtime effectively readonly for the duration of the replay and turns the bypass case into an attributable telemetry event (default) or a fatal load failure (opt-in), with the opt-in flag as the production rollout knob. ## Risks & tradeoffs - **`baseLogger` reshape (incidental).** `ContainerRuntime.baseLogger` is now built via `createChildLogger` with always-on `error` properties (`inStagingMode`, `isApplyingStashedOps`, `isReadOnly`). Every error event flowing through `baseLogger` — including downstream consumers — now carries these tags. Behavior change for telemetry pipelines reading raw `baseLogger` output. - **Log-only by default; throw is opt-in.** `Fluid.ContainerRuntime.EnableSubmitDuringStashedApplyThrow` is off by default — a bypass surfaces a `SubmitDuringStashedOpApply` error event but does not close the container. Setting the flag opts in to the fatal `closeFn` + throw shape. The default favors observability over forcing a load failure in production for a DDS that quietly bypasses the readonly gate. ## Test plan - [x] PSM lifecycle tests cover eager open, no-op without stashed state, full-drain close, stay-open across partial drains, the close hook firing exactly once across repeated `applyStashedOpsAt` calls, and lifecycle staying `"applying"` when the final apply throws. - [x] ContainerRuntime submit-during-apply tests: default path logs the error event and does not throw or close; `EnableSubmitDuringStashedApplyThrow=true` throws + closes + logs; `BlobAttach` allowlisted (no throw); no throw outside the apply window. - [x] End-to-end test: with `EnableSubmitDuringStashedApplyThrow=true`, a `valueChanged` listener that does a cross-map edit during apply causes the load to reject with the expected `UsageError`. - [x] Existing PSM "has both pending messages and initial messages" test updated to push directly to the private queue (the prior flush-while-applying path is now a usage error). - [x] Container-runtime mocha suite passing locally, biome format clean, eslint clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) — signed: Claude (Opus 4.7) --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>