Tecnix dep tracking + eval caching

[joshheinrichs-shopify]

robo summary 🤖

The high-level goal is threefold: speed up evaluation in WCB,
support dependency graph generation for the merge queue, and decouple
Tecnix from Tectonix's zone business logic. WCB primarily needs fast target
discovery and selective target evaluation. The merge queue needs to know which
targets are affected by which source changes so it can safely handle concurrent
merges without serializing everything or rebuilding the world. Tecnix should
provide evaluator primitives for these workflows without embedding
Tectonix-specific knowledge about zones, modules, or target construction.

The interface is split around those caller needs. builtins.tecnixTargetNames
supports WCB's discovery step: it returns a flat list of fully-qualified target
names such as //areas/tools/dev:__devEnv without forcing target values.
builtins.tecnixTargets supports WCB's evaluation step: given explicit target
names, it calls the resolver and returns one value per target in input order.
builtins.tecnixDependencies supports merge-queue dependency analysis: it
evaluates selected targets under tracking and returns the repo-relative source
inputs that influenced each target.

The resolver contract is the boundary between Tecnix and Tectonix. Tecnix
imports a repo-relative resolver such as system/tectonix/resolve.nix, calls
it with { system = ...; }, and expects something like:

{
  resolve = target: ...;
  allTargetNames = [ "//areas/tools/dev:__devEnv" ... ];
}

Everything zone-specific stays behind this contract. Tecnix does not need to
know how zones are discovered internally, how zone.nix is interpreted, or how
Tectonix modules construct target definitions. This is what lets Tecnix stay
focused on evaluation, caching, tracking, and source access.

For WCB, the normal flow is: discover names, select targets, evaluate those
targets. Discovery must be cheap because it happens at world scale. Evaluation
must be selective because constructing every target is too expensive. This is
why tecnixTargetNames intentionally avoids resolving target bodies, and why
tecnixTargets takes an explicit targets list instead of implying "all
targets."

For the merge queue, the important artifact is the dependency graph. Example:

{
  "//areas/tools/dev:__devEnv" = [
    "system/tectonix/resolve.nix"
    "system/tectonix/lib/resolve.nix"
    "areas/tools/dev/zone.nix"
    "areas/tools/dev"
  ];
}

Given a merge that changes areas/tools/dev/zone.nix, the queue can identify
affected targets. Given a merge that touches unrelated files, it can avoid
unnecessary evaluation/builds. This is what makes concurrent merges tractable:
the queue can compare changed paths against target dependency closures instead
of serializing all merges or rebuilding everything.

Dependency tracking happens at the source-accessor boundary, not by statically
analyzing Nix. During tecnixDependencies, Tecnix installs a per-target
tracking context. When evaluation imports files, reads source paths, or
fingerprints source directories, the accessor records repo-relative paths. This
captures resolver imports, Tectonix library imports, zone files, and source
directories in the same format as Git changed paths.

Dirty worktrees are first-class source state. Tecnix evaluates against a clean
Git tree plus an optional checkout overlay. Clean paths fingerprint as Git
blob/tree SHAs. Dirty checkout paths are read from disk, and their content is
folded into the fingerprint. This means local development behaves like committed
source from the dependency tracker's point of view: dirty resolver files, dirty
zone.nix files, or dirty children under tracked source directories all change
the source closure and prevent stale cache reuse. It also means tecnixTargets
sees local edits without requiring a commit.

Directory inputs are tracked coarsely. For a pattern like src = ./., Tecnix
records the directory path and fingerprints it by Git tree SHA rather than
recording every child file. That keeps dependency lists compact while still
invalidating correctly when any child file changes, because the directory tree
SHA changes. With a dirty checkout overlay, a dirty child under that tracked
directory augments the directory fingerprint, so we still avoid listing every
child while remaining correct for local edits.

The cache is source-closure based so it works across commits. It does not care
whether the overall commit SHA changed; it cares whether the tracked inputs for
a target changed. Conceptually, a cached source closure looks like:

{
  "target": "//areas/tools/dev:__devEnv",
  "resolver": "system/tectonix",
  "system": "aarch64-darwin",
  "pathFps": {
    "system/tectonix/resolve.nix": "git:4b825dc642cb6eb9a060e54bf8d69288fbee4904",
    "system/tectonix/lib/resolve.nix": "git:9af3b7d7c2a53477e8b99d3a51a78e1c6d12abcd",
    "areas/tools/dev/zone.nix": "dirty:sha256:abc123...",
    "areas/tools/dev": "git:3e7f91ac7b13f7c1e61b0e6e8f2a0c1b22222222;dirty=sha256:def456..."
  }
}

The public dependency list is the set of paths in that source closure. For
files, the fingerprint is the Git blob SHA when clean, or a dirty content
fingerprint when overlaid from the checkout. For directories, the fingerprint
is the Git tree SHA plus dirty child state when relevant. If two commits have
the same fingerprints for the tracked paths, the dependency result can be
reused across those commits.

When multiple source closures are retained for a target, they can be organized
as a trie to reduce Git ODB lookups. Each edge is a path plus expected
fingerprint. Lookup fingerprints only enough paths to distinguish candidate
closures:

root
└── system/tectonix/resolve.nix = git:4b825...
    └── areas/tools/dev/zone.nix = dirty:sha256:abc123...
        └── areas/tools/dev = git:3e7f...;dirty=sha256:def456...
            └── cached dependency list

If an early fingerprint does not match, Tecnix rejects that candidate without
checking the rest of the closure. This matters when there are many retained
closures for the same target across different source states.

The current limitations are mostly around interface and storage.
builtins.tecnixDependencies is useful for explicit target sets, but
whole-world dependency graph generation may be cleaner as a Tecnix-specific
CLI/evaluator side output because making "the graph of all targets" itself a
target creates self-reference problems. The cache is also still stored in Nix's
generic SQLite fetcher cache as JSON-ish values, and bounded retention like "50
closures per target" is a crude scaling knob. Long term, this should become a
real dependency index with explicit schema, path-prefix lookup, GC/versioning,
and first-class support for answering: "given these changed paths, which
targets are affected?"

Future work: the dependency graph can support developer tooling beyond the
merge queue. For example, a shell hook could compare the current checkout
against the dependency closure of the active dev environment and warn: "your dev
environment is stale; please run dev up." This should be treated as a
consumer of the graph, not the core reason for the evaluator changes.

[EsterKais]

Approving this. The design is clean, and the tests are genuinely great — they
check the hard stuff (shared values, negative lookups, dirty files, symlinks,
cross-commit invalidation), and they compare a target evaluated on its own against
the same target in a shared and a parallel run.

I'm approving rather than asking for changes first, and that's on purpose. Merging
this into tecnix doesn't actually change anything in World — we pin a specific
tecnix commit, so nothing moves until a separate bump PR. And everything I raise
below only matters on the dependency-graph path, which isn't driving any real
decision yet. Normal builds never touch the cache. So I'd rather land this now and
handle these as follow-ups before we turn the dependency graph on, instead of
growing the PR.

A few things my agent raised that we discussed — they seemed like concern points,
or at least clarification points, for me:

1. A .gitattributes change can fool the cache.
Git has a way to say "pretend this file isn't here" (export-ignore, via
.gitattributes). The cache remembers each file it read plus a fingerprint of
that file's contents. The catch is the fingerprint only tells us whether the
file's contents changed — it says nothing about whether the "pretend it isn't
here" rule changed. So if someone adds a rule that hides a file a target read, the
file's contents haven't changed, its fingerprint still matches, and the cache says
"nothing changed" and hands back a stale answer — even though a real re-run can't
see that file anymore.

In the code: getTecnixRepoAccessor turns on export-ignore but doesn't pass the
attribute info the way the zone accessors do in makeZoneAccessorOptions, and
GitPathFingerprintSourceAccessor::getFingerprint only records the raw object id
and mode, nothing about attributes.

The reassuring part: it's only risky in one direction (hiding a file that was
read; un-hiding is already caught), and some cases get caught by a directory
fingerprint. What are your thoughts?

2. Every value gets a little bigger the moment we bump World.
Merging here is harmless, but once World runs on this evaluator, every value is 8
bytes bigger (24 → 32), for all evaluation, whether tracking is on or not — plus a
small check when values finish and a branch in the force path. My worry is memory
and cache pressure on big evals, since value size is exactly the thing upstream
works hard to keep small. Is this something we need to worry about?

3. A shared function can quietly lose its file list — nice-to-have safety net, not a blocker.
accessSetForCopiedValue deliberately drops a function's file list when it's
reused and carries more than one file, so shared helpers don't smear their whole
history onto every target. I think that's the right call and I wouldn't change it.
The catch is it depends on resolver authors remembering to wrap file-reading
shared helpers in a scope, and nothing catches it if they forget. Instead of
changing the drop, could we add a CI test that evaluates targets both on their own
and in a shared/parallel run and diffs the results? A forgotten scope would show
up as the two runs disagreeing — so it turns "hope nobody forgets" into an
automatic check.

4. Just checking on the one lock.
The access-set graph uses a single lock for interning and for merging into each
target's total. The common operations stay off it, and the expensive flatten
happens after the workers finish, so I'd guess it's fine — but under heavy
parallel tracked eval, is that one lock going to hold back scaling, or is it
clearly off the hot path? Just want your read, not asking for a change.

These are all follow-ups or questions, not blockers — landing the foundation makes
sense to me.