Niklas Mischkulnig

Turbopack: emit/collect (#96631) Only add this for using it for Server Actions for now Undocumented, not API stable. But we can't really put it behind a flag, because the next-custom-transform is going to add it New traits: ```rust /// A module that can collect other modules during the collect phase. #[turbo_tasks::value_trait] pub trait CollectingModule: Module { /// The namespace that this module is interesed in #[turbo_tasks::function] fn namespace(self: Vc<Self>) -> Vc<RcStr>; #[turbo_tasks::function] fn as_chunk_item( self: Vc<Self>, module_graph: Vc<ModuleGraph>, chunking_context: Vc<Box<dyn ChunkingContext>>, entry_chunk_group: Vc<Modules>, ) -> Vc<Box<dyn ChunkItem>>; } #[turbo_tasks::value_trait] pub trait EmittedModuleReference: ModuleReference + ValueToString { #[turbo_tasks::function] fn data(self: Vc<Self>) -> Vc<CompileTimeDefineValue>; } ``` Example: ```javascript __turbopack_emit__('./c.js', { namespace: 'my-test', data: 'data-for-c', // optional exports: ['c'], // optional with: ... // optional, import attributes like with ESM }) // ------------------------- const getList = __turbopack_collect__({ namespace: 'my-test', }) const list = getList(); ``` Shortcomings in the current version (which are not a blocker for now however): - data-only emit not implemented - stylegroups aren't properly implemented. emitting styles is probably broken - pages router is still a problem: it has two separate ChunkGroup::Entry, so you only see what was emitted on the current runtime, not both Recreation of https://github.com/vercel/next.js/pull/91100

Turbopack: cross-module constants (#90300) Closes https://github.com/vercel/next.js/issues/92082 This is now a proper compile-time constant: ```js import { IS_DEV } from './other' if (IS_DEV) { // statically evaluates to `true` console.log('x') } else { require("library") // not bundled } console.log(IS_DEV); // is replaced with console.log(true); ``` ```typescript // other.ts export const SOME_VALUE = 'x' const node_env = process.env.NODE_ENV const development_ent = 'development' export const IS_DEV = node_env === development_ent ``` You can use code to compute constants just fine, and use any existing constants such as `process.env.NODE_ENV`. Currently, you can't use imports to other constants modules, but we can add that later on. We can't perform this constants check for every single import, so you need to either - have `UPPER_CASE` import names as in the example above - or use `import { lower } from './other' with { turbopackConstants: 'true' }` Then that referenced module will be analyzed for constants, and if the referenced export is a constant, it will participate in constant inlining just as `process.env.FOO`. This also works fine with barrel imports, you can still do `import { IS_DEV } from './barrel.js'` and it will find the `constants.js` file which in itself will indeed only have constants exports. Because it's not always opt-in at the import site, we can't automatically make non-constant exports an error. For that you have to add `'use turbopack: constants'` at the top of the module, which will make it an error if any constant import references that module, and the module has any non-constant exports: ``` error - [analysis] /turbopack/crates/turbopack-tests/tests/snapshot/comptime/cross-module/input/other.constants.js:8:7 Export NO_CONSTANT is not a constant 4 | const development_ent = 'development' 5 | 6 | export const IS_DEV = node_env === development_ent 7 | + v--------------------------------v 8 + export const NO_CONSTANT = globalThis.foo + ^--------------------------------^ 9 | It was analyzed to be FreeVar(globalThis)["foo"] Import trace: test: ./turbopack/crates/turbopack-tests/tests/snapshot/comptime/cross-module/input/other.constants.js ./turbopack/crates/turbopack-tests/tests/snapshot/comptime/cross-module/input/index.js ``` Some prior art: https://rspack.rs/blog/announcing-1-5#const-inline-optimization, https://rspack.rs/config/optimization#optimizationinlineexports No compile-time impact on a big app: ``` canary dfbc3dc6b7: 438.11s user, 68.35s system, 801% cpu, 1:03.22 total 439.44s user, 71.47s system, 756% cpu, 1:07.56 total 440.32s user, 68.57s system, 750% cpu, 1:07.81 total constants 2d7eb8220d298133212813b7267a5847eed03a22 433.22s user, 65.44s system, 800% cpu, 1:02.32 total 440.52s user, 67.68s system, 770% cpu, 1:05.95 total 433.13s user, 69.47s system, 776% cpu, 1:04.74 total ``` a8620556af64ef550373cca1e1fc701a1a8583cc

More granular cache keys for use-cache entries (#95233) Caveats: - I had to skip `react[-dom][/*]` and `private-next-rsc-server-reference` and `private-next-rsc-cache-wrapper` imports in the code hash and env-var tracking. Because those all end up pulling in app-page-turbo.runtime.prod.js which reads many env vars and would cause constant deopting. But this should still be correct. The code of these imports is included via the Next.js version, and no env vars should change the semantics of any of those imports. - Static env var reads are collected, but too dynamic accesses are silently ignored and don't lead to deopts (same goes for env var reads in native NAPI addons). This means that this cache reuse is not guaranteed to be 100% guaranteed to never lead to stale caches. - Code hashing and env var collection happens on a per-module basis. So if you put all use-cache functions into a single file and/or together with react components, then you will see extraneous invalidations. This will be fixed by either generally enabling module splitting for all of Turbopack, or by adding a special transform that does it for use-cache functions. Followups: - There are some env var static analysis gaps that will be immediate followups before broader testing. These are the various TODOs added in https://github.com/vercel/next.js/pull/95310 - Client components invalidation is very coarse grained right now (statically imports any client component anywhere -> deopt completely). Followup for the future. But the current setup is correct. This would just improve effectiveness further - Do this in dev as well. Currently there is no NFT at all in dev (for performance reasons) - Module splitting for more granular tracking - Include entropy when serializing server reference arguments for cache key Todo: - [x] Use implementation code hash (includes inlined env vars): from #94234 - [x] Include non-inlined runtime env vars: from #95310 - [x] Include client reference manifest (very coarse for now) - [x] Include Next.js version (for wire format, etc) - [x] This is now done for all use-cache entries now. Not just for `use cache: remote`. Is that the intended behavior? Yes - [ ] ~~if `NEXT_DEPLOYMENT_ID` is in the env vars. just deopt and don't care about stringifing and hashing the env vars~~ - [x] Is cache key size a problem? Currently you can get this: (values are always hashed) `CustomCacheHandler::get ["80e6f6560092f0078775e7e787c1c10ecf6dea0bc4",[],["d984fbaa996274737f3b59345a300a20","16.4.0-canary.5","__NEXT_NO_MIDDLEWARE_URL_NORMALIZE=undefined","NEXT_OTEL_PERFORMANCE_PREFIX=undefined","__NEXT_PRIVATE_ORIGIN=a04f4b9d6a8f42724740a480e9a2bc67c053dc04377be831c0e2c407a1422004","NEXT_PRIVATE_RESPONSE_CACHE_TTL=undefined","NEXT_PRIVATE_RESPONSE_CACHE_MAX_SIZE=undefined","__NEXT_CACHE_COMPONENTS=b5bea41b6c623f7c09f1bf24dcae58ebab3c0cdd90ad966bc43a45b44867e12b","__NEXT_ROUTER_BASEPATH=undefined","__NEXT_MANUAL_CLIENT_BASE_PATH=undefined","__NEXT_INSTRUMENTATION_CLIENT_ROUTER_TRANSITION_EVENTS=undefined","__NEXT_APP_NAV_FAIL_HANDLING=undefined","__NEXT_GESTURE_TRANSITION=undefined","__NEXT_USE_OFFLINE=undefined","NEXT_DEBUG_BUILD=undefined","__NEXT_VERBOSE_LOGGING=undefined...]] [["_N_T_/layout","_N_T_/page","_N_T_/","_N_T_/index"]]`

Turbopack: expose list of non-inlined env vars (#95310) Add a new method to `EcmascriptAnalyzable` to return the list of non-inlined env vars In this PR, this isn't used anywhere outside of the tests yet While `cargo bench -p turbopack-ecmascript` does regress: ``` Benchmarking references/packages-bundle.js/full: Warming up for 1.0000 s references/packages-bundle.js/full time: [152.99 ms 153.34 ms 153.70 ms] change: [+11.939% +12.678% +13.307%] (p = 0.00 < 0.05) Performance has regressed. Found 4 outliers among 100 measurements (4.00%) 4 (4.00%) high mild Benchmarking references/packages-bundle.js/tracing: Warming up for 1.0000 s references/packages-bundle.js/tracing time: [153.07 ms 153.92 ms 155.15 ms] change: [+34.997% +36.345% +37.554%] (p = 0.00 < 0.05) Performance has regressed. Found 4 outliers among 100 measurements (4.00%) 2 (2.00%) high mild 2 (2.00%) high severe ``` There is no measurable overall perf impact: ``` commit 7d29e358c8ac383a3019d552fd54f4d05645bf39 (HEAD, tag: v16.4.0-canary.5) v16.4.0-canary.5 pnpm next build --experimental-build-mode=compil 400.89s user 51.96s system 747% cpu 1:00.61 total pnpm next build --experimental-build-mode=compil 394.99s user 54.10s system 757% cpu 59.289 total pnpm next build --experimental-build-mode=compil 399.87s user 58.71s system 772% cpu 59.398 total ``` vs ``` commit 9335597135118b8332b8e0d5a3bcedb596a3ef07 (HEAD -> mischnic/env-var-references) remove runtime_all pnpm next build --experimental-build-mode=compil 399.83s user 50.09s system 719% cpu 1:02.57 total pnpm next build --experimental-build-mode=compil 403.57s user 53.79s system 743% cpu 1:01.52 total pnpm next build --experimental-build-mode=compil 402.23s user 55.05s system 758% cpu 1:00.29 total pnpm next build --experimental-build-mode=compil 400.17s user 55.15s system 761% cpu 59.817 total ```

Bump swc to v77 (#97407) There have been breaking changes to the AST: - `BlockStmtOrExpr` is now `ArrowFunctionBody` - function bodies are `FunctionBody` instead of `BlockStmt` - JSX text literals are `Wtf8Atom`, instead of `Atom`

Change `loadManifest` to return undefined with `handleMissing` (#96530) Previously, this returned `{} as T` which was unsafe. Now it properly returns `undefined` instead and the caller can decide what the fallback value should be

Turbopack: constant evaluate `x in y` (#95286) Prerequisite for https://github.com/vercel/next.js/pull/95233 Constant evaluation of ```js if (!('NODE_ENV' in process.env)) { console.log('existing') } console.log('NODE_ENV' in process.env) ``` -> ```js if ("TURBOPACK compile-time falsy", 0) //TURBOPACK unreachable ; console.log(("TURBOPACK compile-time value", true)); ```

Respect NEXT_HASH_SALT for server side assetsHashes (#95738) Also respect the hash salt for `assetsHashes`

Turbopack: compute code hash per `use cache` function (#94234) This computes the implementation hash of a given `use cache`: - it includes the module containing the `use cache` function, plus all transitive imports of that module (including externals via NFT) - what does it hash - for bundled code, it hashes the generated code (so including any AST transforms, inlined env vars, etc) - for NFTd files, it hashes the file content on disk - So this is basically equivalent to hashing the generated JS chunks, except that it's scoped to only what the given actions module uses - (so notably, if you have multiple use cache functions in a given file, then they will all have the same hash) <br/> - [x] Gate behind experimental flag - [x] Only compute hash for `use cache` functions, not all server actions - [ ] `final_read_hint` is a problem. Some chunk items are now codgened twice, so it recomputes the AST - [x] disable for dev for now - [ ] Measure perf impact ```jsonc // .next/server/server-reference-manifest.json { "node": { "806f4954cfbb75404a19d6d405065ed9059cc0cab2": { "workers": { "app/rsc/page": { "moduleId": "[project]/bench/app-router-server/.next-internal/server/app/rsc/page/actions.js { ACTIONS_MODULE0 => \"[project]/bench/app-router-server/app/rsc/logic.js [app-rsc] (ecmascript)\" } [app-rsc] (server actions loader, ecmascript)", "async": false, "exportedName": "$$RSC_SERVER_CACHE_0", "filename": "bench/app-router-server/app/rsc/logic.js", "codeHash": "413e394d9597b35df12828a6a7fd6363" // <------------ } }, "filename": "bench/app-router-server/app/rsc/logic.js", "exportedName": "$$RSC_SERVER_CACHE_0" } }, "edge": {}, "encryptionKey": "jOF2TbpNLjp2oOhl3VPBwGE7luodnei9clJPq/gaYQo=" } ```

Migrate from box_patterns to deref_patterns (#97924) TLDR: just enable `deref_patterns` and remove the `box` syntax. And it just works `box_patterns` will be removed entirely in the next Rust nightly: https://github.com/rust-lang/rust/pull/156749