Andrew Clark

Port React's @gate test directive to the e2e harness (#96228) The test suite has accumulated a bunch of patterns for disabling tests that are known to fail under some configuration: `it.skip`, `if (isNextDev) { test('skipped in dev mode', () => {}); return }`, whole describes toggled off by checking `process.env.__NEXT_CACHE_COMPONENTS`. These all have the same flaw: nothing tells you when the thing you skipped starts working. The test stays disabled forever, and the workaround it was guarding rots along with it. React solves this with the `@gate` pragma, and this PR ports it to the Next.js e2e harness: ```ts // Blocked on the optimization that marks a route as fully static when // no dynamic params are referenced in Server Components. // @gate !cacheComponents it('navigates to a page with a lazily-generated static param', async () => { // body unchanged }) ``` The test still runs. If the condition is false and the test fails, the failure is absorbed and the suite stays green. If it _passes_, the suite fails: the gate is stale, delete it. So instead of a skip that hides a fixed bug indefinitely, you get a CI failure the day the fix lands. When the condition is static, the inversion is Jest's own `test.failing` under the hood. A lazy condition isn't known until the fixture's resolved config is read inside the body, so those tests invert at runtime instead. `// @force-gate <condition>` skips for real — for tests that can't even be attempted (prefetching is disabled in dev, deploy has no local build output, the fixture won't build under the condition), and for tests of a new API, where the disabled state can only throw and running it proves nothing: ```ts // Prefetching is disabled in dev, so this suite has nothing to test. // @force-gate prefetching describe('segment cache prefetch scheduling', () => { // ... }) ``` There's no staleness check in that case, so this is a judgment call: prefer `@gate` when the off state fails for a meaningful reason — the flag changes behavior that already exists — and `@force-gate` when the body can only throw because the API doesn't exist. A static condition (mode, bundler) resolves at collection time into a normal Jest skip. A lazy condition resolves at runtime, and when a lazy force-gate on a describe is false, we skip the fixture build entirely — that's what makes it usable for suites whose fixtures are build-incompatible with the condition. (One caveat: Jest has no way to skip a test that's already running, so these report as passing with a warning in the log, not as skipped.) Conditions live in a hand-written registry. I considered deriving the lazy ones from the config schema automatically, but a gate is a claim about which dimension of the test matrix explains a failure, and I'd rather each of those claims be spelled out with a description. Referencing an undeclared name fails the suite at collection time, so a typo can't silently disable a gate. The important design decision for lazy conditions is that they read the fixture's _resolved_ config, never `process.env`. The env var isn't the truth: `__NEXT_CACHE_COMPONENTS=true` only applies when the fixture doesn't set `cacheComponents` itself, and config resolution implies flags the fixture never mentions (`cacheComponents: true` alone turns on `experimental.ppr`). Resolution happens in a child process, because in-process `loadConfig` would leak the fixture's `.env` files into the Jest worker. Suites with no lazy gate never pay for any of this. The condition expression is parsed using a small grammar (also ported from the React repo). An expression that doesn't parse fails the suite: ```ts // @gate mode === 'start' && !cacheComponents // @gate !(turbopack || rspack) ``` There's also a runtime version, mirroring React's `gate(flags => flags.enableFoo)`, for tests that run under both states but assert differently (and for `it.each`, where the pragma can't attach): ```ts import { gate } from 'next-test-utils' it('renders the fallback', async () => { if (await gate((conditions) => conditions.cacheComponents)) { // PPR: the fallback is part of the static shell } else { // fully dynamic: the fallback streams in } }) ``` It also accepts the pragma expression language as a string: `await gate('cacheComponents && !dev')`. Docs are in `test/lib/gate/README.md`; `test/unit/gate/` covers the transform, the expression language, and the runtime.

Attempt static prefetch before resorting to runtime (#96095) If we're reasonably confident that a segment can be prefetched statically without omitting data that would have been included during a runtime prefetch (e.g. cookies), the client should attempt prefetch the segment statically instead of going straight to a runtime request. The decision for whether to do this is based on the ShouldAttemptStaticPrefetch added in previous steps. If it turns out the static response is not sufficient, then it will fall back to a runtime request. This makes prefetching cheaper for pages that are fully statically renderable. We can make the optimization better in the future with more reliable per-segment computation of the ShouldAttemptStaticPrefetch, but the current approach should at least work for fully static pages, which is what's most important. This optimization applies during both the Shell phase and the Speculative phase of the prefetching algorithm.

[Flight] Add 'pending_weak' to Flight thenable protocol (#37154) Added behind a new experimental flag, `enableFlightWeakThenables`. Adds a new thenable status to the Flight protocol: `'pending_weak'`. Unlike a regular pending thenable, a weak thenable does not block the stream from closing. If it settles while the stream is still open, its value is emitted like a normal pending thenable. Otherwise its reference is left unfulfilled and on the client it stays forever pending, without erroring, even when the connection closes. It's up to the client to handle the unresolved promise in an appropriate way. The motivating use case is being able to encode metadata about a Flight stream into the response itself. For example, a framework might want to track whether a page varies by search params. It could represent this in the response as a `Promise<boolean>` that resolves to `true` as soon as the component being rendered in the stream accesses search params. If the thenable never resolves by the time the stream closes, then the client knows that no search params were ever accessed. In the future we could add a higher-level API for encoding this kind of information. For now, we intentionally start with the low-level primitive so frameworks can experiment in userspace without adding significantly to React's surface area. Internally, Flight already uses its own private thenable statuses, like `'resolved_model'`, and the protocol is designed to treat any status besides `'fulfilled'` and `'rejected'` as equivalent to `'pending'`, so `'pending_weak'` slots into the existing machinery. On the wire, a weak reference is encoded as `$w<id>`, next to `$@<id>` for regular promises, so the client knows its row may intentionally never arrive. On the client, a weak reference behaves like any other pending promise until the response closes; then, instead of erroring, it is left forever pending.

Unify full/partial navigation response types (#96439) Introduces a new type, TransportData, to replace FlightDataPath. The new format is the same regardless of whether the entire page is rendered by the server or if some segments are intentionally omitted. The type also unifies FlightRouterState and CacheNodeSeedData into a single tree that includes both route information and RSC data. Previously these were sent in two separate trees that were isomorphic by convention, requiring the consumer to walk both in parallel. These new types are intended to be transport formats only. The client already has its own, richer data structure called RouteTree. The transport format is converted into the client format at the network decoding boundary. This does not remove FlightRouterState from the codebase entirely; it's still used both for issuing requests to the server (Next-Router-State-Tree) and for tracking state in the browser's history object. A future refactor may replace it for the purpose of issuing requests, but it will likely survive as the format for tracking state in history. This PR does not yet update the server to produce TransportData directly; a temporary adapter layer is used to convert from the old data formats to the new format. This intermediate step will not land on its own; the next PR in the stack will both update the server and delete the temporary adapter layer.

Convert per-segment prefetches to NavigationFlightResponse format (#96877) This is the last of the changes to convert all RSC requests to a single unified response type, NavigationFlightResponse. The old format was a positional array whose ordering had to be kept in sync by hand between the server and the client. The new format is the same root-anchored tree used by every other response, so the dedicated client write path for per-segment responses is deleted and these responses go through the shared one. A few functions are renamed as their roles change: convertServerPatchToFullTree becomes createNavigationSeed (it now also accepts trees with no base to overlay), fetchSegmentsOnCacheMiss becomes fetchSegmentPrefetchesUsingStaticRequest (the counterpart of fetchSegmentPrefetchesUsingRuntimeRequest), and writeSeedDataIntoCache becomes writeTreeDataIntoCache (seed data no longer exists as a concept). During the unification, a few inconsistencies and oversights were caught and fixed: response data with no matching pending entry is written as a detached entry instead of dropped, and a response's byte size is spread across the entries it actually fulfilled rather than the old chain-based count. The fallback retry loop also marks itself pending before the first cache write instead of after, closing a window where the scheduler could spawn a duplicate revalidation. The second commit unifies cache keying for prefetched segments. There were two implementations of the logic that decides which cache key a prefetched segment is written to, one for per-segment responses and one for responses produced by a live render. Now there's one: if the server reported which params the segment depends on, key the entry using those; otherwise fall back to the keying implied by how the payload was fetched. A segment is only treated as param-independent when the server says so. Two precedence fixes were needed to merge them, and a few other inconsistencies found during the unification are fixed along the way; see the commit message for details.

Track whether runtime data is accessed during prefetch (#95964) Adds a server-computed signal to static per-segment prefetch responses that tells the client whether a runtime prefetch request would return more content than the static response already contains. The signal is computed by tracking whether the prerender accessed any data source that hangs during a static prerender but would resolve during a runtime prerender. For example: cookies, headers, fallback params, and search params. The information is encoded two places: - As a prefetch hint called ShouldAttemptStaticPrefetch. This tells the client that it's worth attempting to do a static prefetch instead of a runtime one. It's only an optimization, though: if the static response ends up being insufficient, the client will follow it up with a runtime request. It's semantically OK if there are false positives. - Embedded in the static segment response. This tells the client whether the static response is missing data that would have been included in a runtime response. Unlike the prefetch hint, this value must never falsely claim that no runtime request is needed. A follow-up will update the client to skip the runtime request using this information. This commit only encodes the signal into the response.

Fix: Optimistic routing bugs leading to repeated prefetch loops (#97128) Fixes a few bugs related to optimistic routing. Originally reported as a next-intl request waterfall in MarkBekooy/prefetching-request-waterfall-bug#1, then extracted into an isolated regression test: A proxy that rewrites every URL to inject a leading path segment (`/one/two` → `/en/one/two`, i.e. i18n with the default locale hidden from the URL) plus a fully dynamic target route like `/[locale]/[...pages]` leads to an infinite prefetch loop that never resolves. This regression uncovered several oversights in the optimistic routing implementation: when receiving a prefetch response, we did not check whether the response matched the expected result. If the tree mismatched, in some cases the prefetch task would fall into a loop, repeatedly attempting to fulfill the missing data. Now when this happens, we record on the local route definition that a dynamic rewrite occurred, disabling further attempts to optimistically resolve the route. This is the same strategy we were already using for normal navigation responses, now applied to the prefetch path. We also received another bug report with a similar root cause: parallel routes with conflicting dynamic params at the same level (`@modal/[...catchAll]` next to `[username]`) cannot be distinguished using the current traversal algorithm, because it assumes that each segment is resolvable independently without inspecting the children or sibling branches. To work around this issue for now, when this scenario is detected, we disable optimistic routing for the conflicting route. We can model this properly in a future PR by having the server send down the "sibling" dynamic route segments, similar to what we do for static siblings already. Fixes #97135 <!-- NEXT_JS_LLM -->

Serve ISR fallback shells in response to prefetch requests (#94534) Previously, a static segment prefetch that hit a route with an unresolved ISR entry could not be served a fallback shell, so the prefetch cache was left cold until the entry regenerated. This enables prefetch requests to be served the ISR fallback shell immediately, so the prefetch cache can be warmed right away instead of waiting. Because the shell is only a partial response, the client retries the prefetch a bounded number of times so it eventually warms the cache with the full (concrete) response once the server finishes regenerating in the background. Only shells that can actually be upgraded are retried — a route with no generateStaticParams never upgrades, so its shell isn't flagged and the client doesn't waste retries on it. The new serving behavior is gated behind the experimental `appShells` flag, so existing behavior is unchanged when it's off. <!-- NEXT_JS_LLM_PR -->

Convert tree prefetches to NavigationFlightResponse format (#96788) Part of a stack of changes to convert all RSC requests to a single unified response type, NavigationFlightResponse. This step converts static route tree prefetches (i.e. `/_tree` requests). The old format could omit a dynamic param's value from the response so that a statically served response is cacheable across param values; the client parses the value from the URL instead. That behavior moves into the shared decoder, where it's now permitted for any transport response. A cached /_tree response from an older build decodes without a build id and falls back to an MPA navigation, like any other cross-build response. The staleTime field of the old format is not carried over because the client never read it.

Fix repeated navigations while the Instant Navigation lock is held (#95864) With the Instant Navigation Testing lock held (the Navigation Inspector paused), repeated client navigations between pages that share a layout could sometimes get stuck without ever resolving. The root cause turned out to be a subtle problem with the client router's use of `useDeferredValue` to switch between the cached UI and the final UI. The mechanism only works as we intend when mounting a new part of the tree. When updating an existing part of the tree, like a shared layout, React will skip the cached UI entirely. We were already aware of this issue with the `useDeferredValue` approach. The plan is to instead model cached navigation as an optimistic update followed by a transition update. Before we can make such a change, though, we need to make a change in React to allow optimistic updates to suspend (currently, optimistic updates act like sync updates, and can trigger existing Suspense boundaries to switch back to their fallback state). To workaround this issue when using the Navigation Inspector, we're going to change how the inspector works when multiple navigations are performed during the same "lock". Instead of keeping the entire sequence blocked until the end, we will instead only block the most recent one. This more closely matches what the inspector UI implies, anyway: it doesn't show a stack of pending navigations, only the most recent one. This doesn't absolve us from needing to switch to using an optimistic update. That's needed regardless to address similar behavior in production. But this sidesteps the issue for now, and we were planning to make this change to the Navigation Inspector regardless. Co-authored-by: Sam Selikoff <sam.selikoff@gmail.com>