Performance improvement: Switch phantom symlink list for trie#6345
Open
chrisharris-discord wants to merge 3 commits into
Open
Performance improvement: Switch phantom symlink list for trie#6345chrisharris-discord wants to merge 3 commits into
chrisharris-discord wants to merge 3 commits into
Conversation
A symlink on Windows must be created as a "file" or "directory" symlink, and Git can't always tell which until the target appears during checkout; until then it's tracked as "phantom" and rechecked. Phantom symlinks lived in one global list, fully rescanned by process_phantom_symlinks() on every mkdir() (and on every symlink that turns out to point at a directory), making checkout O(n*m) in outstanding phantom symlinks times directories created. Reported in git-for-windows#4059 for a git-annex repo where nearly all symlinks dangle permanently. On a 343-symlink, ~185k-file monorepo fixture, checkout takes 330s and calls process_phantom_symlink() 12,693,504 times -- about 343 * 37016, the directories created. Index phantom symlinks in a trie over their target path's components instead. Waking a path retries only entries registered there or nested under it, so an unrelated directory costs O(1). Same fixture: 54.8s and 343 calls -- one per symlink, no wasted rescans. Also fixes an existing bug: wlink/wtarget are interior pointers into the same allocation as the struct, not separate allocations, so only free(current) is correct; freeing them individually is undefined behavior. Known limitation: a phantom symlink whose target passes through another symlink as an intermediate component may not get woken if that symlink's real target directory is still being populated when it resolves; the next commit addresses this. Signed-off-by: Chris Harris <chris.harris@discordapp.com>
Waking a symlink's own path when it resolves to a directory cascades into nested trie entries, but only once, at that exact moment. If a nested entry's target doesn't fully exist yet then (e.g. a deeper real directory is created moments later), it is never woken again -- the real directory appears under a different trie key than the one the entry is registered under through the symlink. Graft everything registered under the symlink's own path onto its resolved target's trie node before waking both, so a later mkdir() under the real path can still find it. Verified with symlink "T" -> "B/C" through symlink "B" -> "realdir", where "realdir/C" is created after "realdir": T now correctly resolves to a directory symlink, where it previously stayed a file symlink. Signed-off-by: Chris Harris <chris.harris@discordapp.com>
Add a regression test for the grafting fix: a symlink whose target passes through another symlink should still become a directory symlink once its real target appears, even when that target is only populated after the leading symlink has already resolved. Windows path resolution follows a symlink's target regardless of whether it is flagged as a file or directory symlink, so opening a path through it succeeds either way; the type only becomes visible in things like `cmd.exe /c dir`, which marks directory symlinks as <SYMLINKD> and file symlinks as <SYMLINK>. The test asserts on that distinction rather than on read access, since the latter would pass even without the previous commit's fix. Signed-off-by: Chris Harris <chris.harris@discordapp.com>
chrisharris-discord
force-pushed
the
chrisharris/mingw-phantom-symlink-trie
branch
from
July 21, 2026 21:39
07429a3 to
2935e70
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Checkout is O(n·m) on Windows when a repo has many symlinks: phantom
symlinks (whose type as file vs. directory can't be determined until
their target appears) live in one global list, fully rescanned on
every
mkdir(). n = outstanding phantom symlinks, m = directoriescreated. See #4059.
On a 343-symlink, 37,016 directory fixture: checkout takes 330s with
12,693,504 calls to
process_phantom_symlink().Fix
a path only retries entries registered there or nested under it.
Same fixture: 54.8s, 343 calls (one per symlink).
once it converts to a directory symlink, so a symlink nested through
another symlink still resolves correctly.
t2041) for the grafting case.Also fixes a pre-existing bug found while testing:
wlink/wtargetare interior pointers into one allocation, not separate ones, so they
must not be
free()'d individually.Please let me know if the test breaks the 'commits only for windows' section and should be pulled or submitted to git itself!