FluidFramework — Engineering Performance

Switch to React's new JSX transform (#26631)

improvement(build-tools): add explicit return types and handle undefined from findConfigFile (#26186) ## Summary - Add explicit return type annotations to functions and methods across build-tools and bundle-size-tools packages to satisfy the `@typescript-eslint/explicit-function-return-type` rule - Handle the `undefined` case from `TscUtils.findConfigFile()` in repo policy check files, fixing potential runtime errors when config files cannot be found - Fix JSZip import in bundle-size-tools to use namespace import (`import * as JSZip`) instead of named import for compatibility ## Changes **Explicit Return Types** Added return type annotations to ~80 functions/methods in: - `gitRepo.ts` - Git command wrappers - `monoRepo.ts` - Monorepo utilities - `npmPackage.ts` - Package accessors - `timer.ts`, `logging.ts`, `utils.ts` - Common utilities - `buildGraph.ts`, `fluidRepo.ts`, `fluidRepoBuild.ts` - Build system - `leafTask.ts`, `tscTask.ts`, `webpackTask.ts` - Task implementations - `PrCommentsUtils.ts`, `getCommentForBundleDiff.ts` - ADO integration - Various other task and utility files **Undefined Handling** - `fluidBuildDatabase.ts` - Added undefined check for `findConfigFile` result - `fluidBuildTasks.ts` - Added undefined checks in two locations for `findConfigFile` result **API Changes** - `unzipStream` return type updated to use `ReturnType<typeof JSZip.loadAsync>` due to import change --------- Co-authored-by: Joshua Smithrud <54606601+Josmithr@users.noreply.github.com>

test(api-markdown-documenter): Add "deep" config end-to-end test cases (#23586) Adds a test case that leverages new hierarchy options to create a "deep" suite configuration where documents that generate folder hierarchy yield documents named "index" _inside_ their folder,

Introduce Unified ServiceClient API for Fluid Client (#27693) ## Description This add a simple API as a cleaner alternative to fluid-static and aquaduct. The implementation details are well encapsulated, making this a viable alternative to the `fluid-static` public API and a viable successor to the existing much larger legacy API surface we do not want to stabilize. This addresses several issues with our existing APIs: 1. non-legacy (fluid-static declarative model) API: 1. No stable way to use local service for testing (Internally or for customers). Forces tests to use alternative patterns (like test-utils, mocks etc. or a separate azure local service process). 2. No way to avoid bundling in SharedDirectory. 3. Defining and using DataStores is not well supported, especially for the root. 4. Dependency layering is such that its disallowed for DDS packages to use the public API surfaces to do collab tests. 5. No abstraction for Services / ServiceClient makes logic portable over services harder to write. 6. Doesn't support lazy loading DDS and DataStore code (at least not clearly/cleanly). 7. No migration path from legacy APIs to this 3. legacy (encapsulated model, including aquaduct): 1. Leak way too many details with too many non-sealed interfaces presenting tons of theoretical extension points we don't want to stabilize. 2. Encourages use of DataObject and its subclassing based extension approach which is limiting (Like how it makes switching to root trees) hard, and makes changing the implementation without introducing breaking changes very difficult. Because this new "unified" API offers value for our own internal testing (for example in tree), it is useful, even as alpha (or internal). Exposing this new API as alpha should be low risk, as it makes no long-term commitments. Since we have first party in-repo use cases, it should be practical to refine and iterate on this API with use in test and examples before we consider promoting it to beta. Requirements for this new API: 1. Testing: 2. Can easily be used for tests including collab and reopening documents. 3. Can be used by DDS (like shared-tree) tests replacing the need for many of our mocks and test utils 4. Can be used by customers (apps and libraries requiring stable APIs when promoted past alpha). When adopted, customers should not have need of any of our legacy test-utils and mocks (so we can migrate our legacy apps off of our test utils and eventually make them internal only). 5. Will be possible to stabilize to public (or at least beta) without depending on or stabilizing legacy APIs. 6. Is easy to use for tests, including collab. 7. Is easy to use for real production apps (should work the same as with tests). 8. Provides a clear and practical incremental migration path from both existing APIs: there can be gaps we need to close for some use-cases but those gaps need to be practical to close.

(pipelines): Add build performance observability pipeline (#26316) ## Description This PR updates the placeholder pipeline added in https://github.com/microsoft/FluidFramework/pull/26299. The new pipeline does the following: - Collects build metrics from ADO REST APIs for PR and internal builds - Generates an HTML dashboard (published as a pipeline artifact) which includes: - Summary metrics (total builds, avg duration, trend analysis) - Duration trend charts over time - Stage and task duration breakdown charts - Tables of recent and longest builds, including links to the source commits/PRs - Will run on a daily schedule Note: The pipeline must run in the public and internal projects to fetch PR build data and internal build data, respectively. ## Next Steps/Other Considerations - Seek feedback from the team on what other metrics could be useful. - It seems ADO only retains a limited number of PR builds. If possible, we should try to increase the limit. - Consider adding an external data store (especially if we cannot increase the public build limit). ## Misc [Spec](https://microsoft.sharepoint-df.com/:fl:/s/8b2be717-e115-43f1-9d10-bd09d33fd20b/IQAQCryKGC8vSo3_JIdYz-tnAW-5TjJglskciG2A7EQ3jN0?e=A68U9w&nav=cz0lMkZzaXRlcyUyRjhiMmJlNzE3LWUxMTUtNDNmMS05ZDEwLWJkMDlkMzNmZDIwYiZkPWIlMjF2TlJQUTAwMUprQ3BuVmxyYnpzMWoxeE9JU0NSOVFGT2dMOHk2RlpXTkk3YTZqZTc2bksxUWFoTExXaGpYQ3hzJmY9MDFTTE9WT1RRUUJLNklVR0JQRjVGSTM3WkVRNU1NNzIzSCZjPSUyRiZhPUxvb3BBcHAmcD0lNDBmbHVpZHglMkZsb29wLXBhZ2UtY29udGFpbmVy) [AB#55451](https://dev.azure.com/fluidframework/235294da-091d-4c29-84fc-cdfc3d90890b/_workitems/edit/55451) ## Example Screenshot 

build(client): generate eslint 9 configs for all packages (#25989) This change updates the shared eslint 9 config to be TypeScript and updates the script in scripts/generate-flat-eslint-configs.ts to support a `--typescript` flag to output typescript per-package configs instead of ESM. Also adds a `--finalize` flag that will do some additional cleanup once packages are fully migrated to eslint 9. Finally, there are lots of fixes to the script to handle ignore entries, inheritance from other configs, etc. The script now produces configs that work in my testing without any modifications. Even if that doesn't hold true once we are truly migrating (see #25932), merging this beforehand will simplify the review of those changes. All the configs were generated by the script using this command: ``` tsx scripts/generate-flat-eslint-configs.ts --typescript ```

feat(tree): staged allowed types (#25116) ## Description Adds `staged` API to `SchemaFactoryAlpha`. This creates an allowed type in the view schema that may or may not be included int he stored schema. Continuation of https://github.com/microsoft/FluidFramework/pull/24631, with a new PR from my fork so I can have push permissions. Major changes: - Adds SchemaFactoryAlpha.staged - There is no longer a single choice for how to derive a stored schema from a view schema: - Different use cases have been split into different APIs for example (toInitalSchema, toUpgradeSchema) - The underlying toStoredSchema now takes options to control how each staged schema is handled. - Unhydrated content always permits staged content (with the exception of clone): this ensures that export/import round tripping of staged content works. - testDocuments test suite has been expanded so existing round trip tests cover the above. Known issues/limitations: - Recursive types are not supported. Tracked by https://dev.azure.com/fluidframework/internal/_workitems/edit/45711 - Clone should produce nodes with a union of source context and staged types so that both unknown optional fields work, and new staged types can be inserted. Tracked by https://dev.azure.com/fluidframework/internal/_workitems/edit/45725 but also partially hidden by https://dev.azure.com/fluidframework/internal/_workitems/edit/45723 which covers how inserting the out of schema types does not error in unhydrated context currently. This is ok, as we catch them when inserting into hydrated documents and thus the two pugs combine to make the desired behavior, but itn't great, and could cause other issue due to violating internal invariants. Some of the new clone tests cover this. - Some places, mainly those not using alpha typer, can't accept annotated allowed types and thus don't accept staged types. Examples include the root of the tree view config, recursiveObject fields, and likely more. Many of these can be worked around using dedicated alpha APIs and/or wrapping the implicit field schema in an explicit one using SchemaFactoryAlpha.required. Some cases, like recursiveArray do not have a viable workaround, and can be addressed in future work, possibly after stabilizing annotated allowed types.

Add tree-agent package (#25114) This new package hosts the SharedTree semantic editing library. Much of the package config has been copied from other FF packages for consistency (e.g. from dds/tree). The library itself is in a usable state but will be iterated upon heavily in the coming weeks. --------- Co-authored-by: Taylor Williams <60717813+taylorsw04@users.noreply.github.com>

Automatic invalidation based on observation tracker for readers of SharedTree content (#25459) ## Description Adds automatic observation tracking for shared tree content. Adds react hooks to use this, along with strong types for use in those hooks to allow safe passing of nodes in props.

(tree): Incremental summarization of chunked forest (#24949) ## Description [AB#1263](https://dev.azure.com/fluidframework/235294da-091d-4c29-84fc-cdfc3d90890b/_workitems/edit/1263) has an overall description and link to a design document. This PR adds incremental summarization support to chunked forest. Currently, the chunked forest data is encoded and added to a single blob in the summary tree. This change breaks it down into mutiple subtrees where each subtree can be incrementally summarized, i.e., if such a subtree doesn't change between summaries, a summary handle will be added to the summary instead of the entire subtree. The schema will support marking properties as incrementally summarizable (TODO - not in this PR). Any chunks in the fields corresponding to this property will be the boundary at which incrementally summarization will work. Basically, if this property changes or anything within it changes (for objects and arrays), its chunks will be re-summarized, or else, a summary handle will be used. To achieve this, this PR makes the following changes: - Added a new encoded format called `EncodedIncrementalChunkShape` to `EncodedChunkShape` for chunks whose data will be incremental summarized. - Added a new shape called `IncrementalChunkShape` which represents chunks whose data can be incrementally summarized, aka, incremental chunks. - Each incremental chunk is assigned a `ChunkReferenceId` which uniquely identifies it under its parent field. - Added a new field encoder called `incrementalFieldEncoder` which encodes its chunks as `IncrementalChunkShape`. The `ChunkReferenceId`s of all chunks in a field are stored as `EncodedNestedArrayShape` in the encoded data. - Added a new decoder called `IncrementalChunkDecoder` which can decode an incremental chunk. - Added `IncrementalEncoder` and `IncrementalDecoder` interfaces which has properties that help with encoding and decoding of incremental chunks. These are added to the `FieldBatchEncodingContext`. - Added a `ForestIncrementalSummaryBuilder` which tracks chunks across summaries and generates summary tree or summary handles for them depending on whether they change between summaries. - The forest summary tree consists of a blob called `ForestTree` at the root that contains the data for the forest. If there are any incremental fields in the forest, an array of chunk reference ids will be encoded for its chunks in the data. - Fore each incremental chunk, a separate summary tree will be added with its reference id as the key for the tree. - The data for the chunk will be added to a blob called `contents` under the tree. If the chunk has any incremental fields, an array of chunk reference ids will be encoded for its chunks in the data. - The same format is repeated until all incremental fields have been summarized. - Note that currently, an incremental field will have exactly one chunk. This may change in the future where the data may be encoded into multiple chunks. [AB#14308](https://dev.azure.com/fluidframework/235294da-091d-4c29-84fc-cdfc3d90890b/_workitems/edit/14308) [AB#41861](https://dev.azure.com/fluidframework/235294da-091d-4c29-84fc-cdfc3d90890b/_workitems/edit/41861) [AB#41863](https://dev.azure.com/fluidframework/235294da-091d-4c29-84fc-cdfc3d90890b/_workitems/edit/41863) [AB#41864](https://dev.azure.com/fluidframework/235294da-091d-4c29-84fc-cdfc3d90890b/_workitems/edit/41864)