Benjamin Woodruff ‮

Turbopack: Fix missing canonicalization of paths and always use verbatim paths internally for Windows (#95668) I was seeing failures on https://github.com/vercel/next.js/pull/95628 and went down this rabbit hole. - We were (incorrectly) not canonicalizing the `DiskFileSystem` root dir. This happened to work out for us because pnpm also wasn't canonicalizing junction point targets, and so the root dir prefix matched. However after `#95628`, pnpm started canonicalizing junction point targets because we added `pnpm-workspace.yaml` files. This was just coincidence and indicative of much deeper problems. - `dunce` isn't a good fit for us. It picks a win32 or verbatim representation based on path length, which would mean that long paths inside of a short root path would never work because we'd never be able to strip the prefix, since we'd be looking for a win32 path prefix. Omnipath is better for what we want to do: https://docs.rs/omnipath/latest/omnipath/windows/trait.WinPathExt.html - The windows verbatim path format works with long paths (>260 characters) and behaves a lot closer to cross-platform unix paths, which is what we want. E.g. if somebody tries to import a JS module from a Windows 8.3 short path, it should fail. This will fail with verbatim paths (good!). This PR *always* uses the verbatim path format internally within `DiskFileSystem`. - `read_link` was re-inventing `try_from_sys_path` but badly. Just call `try_from_sys_path`. - `turbopack/crates/turbo-tasks-fs/src/embed/file.rs` was dead code. - `validate_path_length_inner` was incorrectly referring to verbatim paths as UNC paths. These are two different things. - Canonicalize any absolute symlink targets outside of the `DiskFileSystem` root path, resolving any symlinks they may have, normalizing case-insensitive paths, and handling Windows 8.3 short name format. ## Some Context About Windows Path Formats: - win32 paths: These are your normal user-friendly `C:\foo\blah` paths. When calling most filesystem APIs, these are limited to 260 characters unless the user has enabled long paths on the system, and avoiding the limit with these paths also requires a [`longPathAware` application manifest](https://learn.microsoft.com/en-us/windows/win32/sbscs/application-manifests#longPathAware). You can mix `/` and `\` in win32 paths, and Windows will normalize the path separator for you. - verbatim paths: This is a rust-specific name (used by the stdlib) for the "extended length" prefixed paths. These are paths starting with `\\?\`. These paths can be up to ~32,767 characters long. They do not support Windows 8.3 shortnames. They do not normalize between `/` and `\`, and do not support `.` or `..` components, however [`Path::join`](https://doc.rust-lang.org/std/path/struct.Path.html#method.join) will do this normalization if given a verbatim path. - Windows 8.3 shortnames: A legacy format where paths can be represented by 8 characters, followed by a tilde and a digit (or hash in some extreme cases where there are many collisions), followed by a 3 character file extension. NTFS supports this, [ReFS](https://learn.microsoft.com/en-us/windows-server/storage/refs/refs-overview) (used by [Windows Dev Drive](https://learn.microsoft.com/en-us/windows/dev-drive/)) does not. - UNC ("Universal Naming Convention"): This is a specific representation of paths used for remote files on network servers. - `file://` URIs: These must be formed from [win32 style paths](https://docs.rs/url/latest/url/struct.Url.html#method.from_file_path), though the normal 260 character limit typically does not apply here. - canonicalize: Corresponds to [`std::fs::canonicalize`](https://doc.rust-lang.org/stable/std/fs/fn.canonicalize.html). This involves actual fs operations (that we can't easily track), and recursively resolves symlinks. It will fail if the file path does not exist. On windows, this always returns a verbatim path. - wide format: Windows paths are utf-16. Generally we don't support paths that can't be encoded to utf-8, but we should try to fail as obviously as we can in these cases. We don't have good filesystem issue handling yet though. - case sensitivity: Generally paths are not case-sensitive on Windows or macOS, but are case-sensitive on most Linux filesystems. The exact definition of how case sensitivity works with unicode paths is highly platform and filesystem dependent. - junction points: The windows equivalent of symlinks (symlinks are also supported, but require developer mode to be turned on). These can contain an arbitrary path reference in any format (win32, verbatim, etc) with any case and with windows 8.3 short names. Junction points are always target absolute paths to directories.

Turbopack: Split up turbo-tasks-fs/src/lib.rs into smaller modules (#96030) This file was way too huge and was wasting a ton of tokens when trying to use it with agents. `turbo-tasks-fs/src/disk.rs` is still a little bigger than I'd like, but I think this is an overall improvement.

Turbopack: Split the read and write codepath data structures for symlinks (#97395) Another attempt at cleanup, similar to https://github.com/vercel/next.js/pull/96955. We need different information to write symlinks or junction points (most notably, if it's a directory or not) than we get when reading symlinks. We were trying to use the same data structure for both, but this lead us to always call `get_type()` as part of reading a link, which was just silly. This also meant that `write_link` had to handle `NotFound` and `Invalid`. `NotFound` is somewhat reasonable (just delete the file), but `Invalid` is awkward.

Turbopack: Add support for specifying additional roots (#98003) Full motivation and plan here: https://app.notion.com/p/vercel/Turbopack-pnpm-Global-Virtual-Store-383e06b059c480579403ddfd71cc2d40?source=copy_link The goal is to allow `DiskFileSystem` to traverse outside of it's own root to other configured `DiskFileSystem`s when following symlinks. We may allow traversal in other situations in the future, but this is limited to symlink resolution for now. ## Global Virtual Store The motivation for this is to enable [pnpm's Global Virtual Store feature](https://pnpm.io/global-virtual-store) (and there are other package managers doing this, including nub and bun). We'd expose the ability to manually configure this in `next.config.js`, but we should also auto-configure ourselves for popular package managers (or at least make a best effort to do so, the `PNPM_HOME` semantics can be complicated). The `ignoreIfMissing` option is provided for this situation: We can configure a bunch of roots automatically, and they only actually get set up if they exist, the check for directory existence is cheap. ## NFT changes This requires a couple extensions to the `*.nft.json` file format: https://github.com/vercel/next.js/pull/98469 ## Related Issues - #93556 - https://github.com/pnpm/pnpm/issues/14972

Turbopack: Add symlinks and additional roots to NFT metadata (#98469) Implements the following extension: ```typescript export interface NftFileList { // File paths relative to the directory containing the `.nft.json` file, or // the current root (if inside of `NftAdditionalRoot`). // // When using webpack, these paths may exist outside of the tracing root. The // [`@vercel/next` package ignores these paths][vc-next]. // // When using Turbopack, these paths are all guaranteed to exist within the // `turbopack.root` specified or inferred from the `next.config.js` file. // // [vc-next]: https://github.com/vercel/vercel/blob/%40vercel/next%404.20.5/packages/next/src/server-build.ts#L1022-L1026 files: string[] // Turbopack extension: A parallel array to `files` (same indices and length) // with file content hashes. For symlinks, the hash of the path of the target // is stored. fileHashes?: string[] // Turbopack extension: Explicit symlink mapping information. // // If included, it's safe to assume that if a file is not in `symlinks` that // it is not a symlink. If this is an empty array, there are no symlinks. // // If omitted, the processor of the nft file must call `read_link` on every // file to determine if it is a symlink and determine the target path. // // This field is always included if `NftJson` includes `additionalRoots`, and // it is always included on `NftAdditionalRoot`. symlinks?: NftSymlink[] } export interface NftJson extends NftFileList { version: 1 // Turbopack extension: A hash of the entrypoint that refers to these traced // files. This hash only depends on the content of the entrypoint file, and // not all of its traced dependencies. entryHash?: string // Turbopack extension: Paths stored with different base paths, typically // outside of the tracing root. additionalRoots?: NftAdditionalRoot[] } // Turbopack extension: A collection of paths stored with a different base path. export interface NftAdditionalRoot extends NftFileList { // Stable unique identifier provided in the `next.config.js`. This can be used // to generate the output path where these files are copied to (e.g. // `.additionalRoots/$[name}`). // // This is guaranteed to use a character set that is valid on most // filesystems, and the identifiers are guaranteed to not have overlaps on // case-insensitive filesystems. name: string // A source path on the build machine that the paths in `files` are relative // to. The final build output directory should not depend on this path. absolutePath: string // Always specified on NftAdditionalRoot. symlinks: NftSymlink[] } // Turbopack extension: Information on a symlink, including which additional // root it maps to. Symlinks that do not cross root boundaries (the common case) // omit the index into `additionalRoots`. // // It is often complicated to transform raw symlink targets to root-relative // paths, and including this information here ensures that the NFT reader gets // the same result that Turbopack's tracing system expects. // // Because the link target type is unspecified, on Windows the reader needs to // call `stat` to determine if a link target is a directory or file. export type NftSymlink = | [ // Index of `files` that refers to a symlink. number, // The target path of the link. In `NftJson`, this path is relative to the // directory containing the `.nft.json` file. In `NftAdditionalRoot`, this // is relative to the current root. string, ] | [ // Index of `files` that refers to a symlink. number, // The target path of the link relative to the specified root. string, // An index into `additionalRoots`, -1 if the target path is relative to // the `.nft.json` file's directory, // // If the symlink target is relative to the same root as the symlink // itself (the "current root"), this field is omitted. number, ] ``` Nothing currently populates `additionalRoots`. https://github.com/vercel/next.js/pull/98003 will do it.

Add a fast-path to 'get_relative_path_to' for a potential common case when 'from' is empty (#98440) When creating an NFT file in https://github.com/vercel/next.js/pull/98003, I store a generic list of "base" paths and then construct the final path in the JSON file as relative to the base path. For most files, these paths are relative to the `*.nft.json` file, but for additional roots, this base path is always an empty string (relative to the root of the filesystem), so being able to re-use the original `RcStr` instead of constructing a brand new string may be beneficial. This also moves the unit tests to be colocated next to the functions that are actually being tested.

Turbopack: Improve how DiskWatcher is configured and fix polling watcher bugs (#96440) - Adds a `DiskWatcherConfig` object to centralize the recursive mode, polling, and invalidation reason options. - **Polling Fix:** Use the non-recursive mode if the polling watcher is selected, ignoring the OS/platform. - **Polling Fix:** Treat mtime events as data update events if the polling watcher is selected (this is what https://github.com/vercel/next.js/pull/96288 was doing, but this PR only does it if polling is enabled) - Only treat the `MetadataKind::Any` event as a data change on macOS. Fixes #80665 Closes #96288

Turbopack: More aggressively debounce filesystem watch events if we detected changes to node_modules (#96116) Previously, we were debouncing update by sleeping 1ms at a time on macos and windows, and 10ms at a time on Linux. During a slow `pnpm install`, or a `git checkout`, this could cause us to do a bunch of extra throwaway work. Changes: - Increase the debounce interval to a consistent 10ms everywhere. This should still be small enough that it's not noticable on macos or windows. - If an event touches `node_modules`, there's a good chance that a package manager is running and many other files will be modified, so extend the batch deadline by 200ms instead of 10ms. - Because there's a chance that the batch deadline could get extended indefinitely (this was always possible, just more likely now) include a compilation event that gets logged after 5 seconds.

Turbopack: Allow DiskWatcher to use a mocked DiskFileSystem, add a small unit test (#96353) Attempting to clean up our testing story here so that we have a way to test the watcher in isolation, ahead of trying to tackle https://github.com/vercel/next.js/pull/96288

[ci] split up large cache-components-errors tests (#95623) See https://github.com/vercel/next.js/pull/95553 for an explanation of why we want to split these up. TL;DR it makes CI faster and saves us some money. In CI we have to run tests at the suite/file granularity, so if a test suite is too long it limits parallelism and if it has to be retried, we have to retry the full suite.