Skip to content

SPIKE: feat(bundler): add native Gemfile.lock parser and resolver#209

Draft
JamesPatrickGill wants to merge 3 commits into
mainfrom
feat/bundler-resolver
Draft

SPIKE: feat(bundler): add native Gemfile.lock parser and resolver#209
JamesPatrickGill wants to merge 3 commits into
mainfrom
feat/bundler-resolver

Conversation

@JamesPatrickGill

@JamesPatrickGill JamesPatrickGill commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

What this does

Adds a new SCA plugin at pkg/ecosystems/ruby/bundler/ that resolves Bundler dep graphs by natively parsing Gemfile.lock — no bundle install, no bundle CLI invocation, no vendor/ or .bundle/ directories created.

The package is self-contained and not wired into the orchestrator in this PR; that lands in a follow-up. The legacy in-CLI plugin at cli/src/lib/plugins/rubygems/ is untouched and continues to serve production scans.

Why

The legacy plugin is bundled inside the cli repo itself (the only ecosystem laid out that way) and uses the @snyk/gemfile npm package to parse Gemfile.lock natively. The principled choice here was a quick audit: every first-party bundler subcommand that emits dep info requires bundle install first — which would mutate the project dir, violate the offline principle, and ship gems into vendor/bundle or the system gemset. The community bundler-graph plugin has the same install requirement.

So native parsing is forced — and Gemfile.lock is plain text with a well-spec'd structure (GEM / GIT / PATH / PLATFORMS / DEPENDENCIES / BUNDLED WITH / RUBY VERSION blocks). The @snyk/gemfile parser is ~150 LoC of straightforward state-machine parsing; porting to Go is low risk.

See the Plugin Audit for the broader ecosystem migration context.

What's in this PR

3 commits, scoped narrowly to the new package:

# Commit What
1 feat(bundler): add native Gemfile.lock parser and resolver types.go (SpecSource, SourceMeta, Spec, Dependency, Lockfile), parser.go (indent-driven state machine, ~295 LoC), depgraph.go (DFS resolver, bundler exclusion, CycleError, BuildOptions hook), plugin.go (SCAPlugin impl, discovery).
2 test(bundler): unit tests for parser, depgraph, plugin 33 unit tests covering parser blocks (incl. git ! stripping), bundler exclusion, diamond/cycle distinction, group plumbing.
3 test(bundler): integration test scaffold with smoke fixtures Build-tag-gated (//go:build integration && bundler) acceptance harness + make test-bundler-integration target. 3 fixtures (simple, with-git-source, with-dev-group) with expected.json goldens. 85.5% integration coverage.

Nothing under pkg/ecosystems/orchestrator/ is touched, so this won't conflict with other in-flight resolver work.

How it works

Native parse of Gemfile.lock using an indent-driven state machine: top-level keywords (GEM, GIT, PATH, PLATFORMS, DEPENDENCIES, BUNDLED WITH, RUBY VERSION) switch the parser mode; indented specs: entries with their transitive dependencies are accumulated per source.

DepGraph construction is a DFS resolver. Two-set cycle handling: visited tracks the ancestor stack (popped on the way back up), added tracks nodes already in the graph (for the diamond / shared-transitive case). A repeat visit while still on the ancestor stack raises *CycleError{Gem, Chain} — matching legacy 422 semantics. TestBuildDepGraph_DiamondNoFalseCycle guards against the classic visited-set bug where shared transitives get miscounted as cycles.

Git ! markers on dep names (e.g. rspec!) are stripped to match legacy. Bundler itself is silently omitted from the dep graph (legacy behavior).

Divergences from the in-CLI legacy plugin

Dimension Legacy (cli/src/lib/plugins/rubygems/) This PR
Implementation language TypeScript via @snyk/gemfile. Go, native parser.
Git ! marker Stripped from dep names. Same.
Bundler itself Silently omitted from graph. Same.
Cycle handling Throws 422 on cycle detection. Throws *CycleError{Gem, Chain}.
Group separation by --dev Not implemented (groups parsed but no flag-based filtering). Plumbed but no-op today — BuildOptions.IncludeDev is wired from options.Global.IncludeDev, but Gemfile.lock doesn't encode group membership. Real group filtering would require parsing Gemfile (not the lockfile) to learn which gems belong to :test / :development groups. Hook left in place for the follow-up.
Root pkg name basename(scanRoot); or path.join(basePackageName, dir) when allSubProjects=true. Same; for nested lockfiles via --all-projects we suffix with the relative dir for distinct root names.

CLI options honored

Option Source Legacy use New plugin handling
--dev / -d options.go:GlobalOptions.IncludeDev Parsed but unused Plumbed via BuildOptions.IncludeDev; no-op until Gemfile parsing lands
--all-projects options.go:GlobalOptions.AllProjects Sets allSubProjects → affects naming Suffixes root name with relative dir
--target-file options.go:GlobalOptions.TargetFile Required input Required input
--exclude-paths global Handled at orchestrator Handled at orchestrator

What's NOT in this PR

  • Orchestrator registration — deliberately deferred to avoid conflicting with other in-flight resolver work.
  • Removal/deprecation of the in-CLI legacy plugin at cli/src/lib/plugins/rubygems/ — that stays put until orchestrator GA.
  • Real Gemfile parsing for true group separation.
  • CircleCI matrix.

A follow-up PR will land the orchestrator registration and (separately) the Gemfile parser for group filtering.

How to verify

# unit tests (no Ruby or Bundler needed — native parser)
go test ./pkg/ecosystems/ruby/bundler/...

# integration tests
make test-bundler-integration

# coverage
go test -cover ./pkg/ecosystems/ruby/bundler/...

Caveats for reviewers

  • --dev group separation: plumbed but no-op. The plan called this out as an intentional enhancement over legacy, but the agent correctly noted you'd need Gemfile parsing (not just Gemfile.lock) to actually filter groups. Hook is in place; full implementation deferred.
  • Root pkg name for nested lockfiles: legacy's allSubProjects naming convention should be cross-checked before merge.
  • Coexists with the in-CLI plugin: this PR adds a new path in cli-extension-dep-graph; the existing cli/src/lib/plugins/rubygems/ keeps running for production scans until the orchestrator FF flips.

Risk

The new package is not registered with the orchestrator, so it has no effect on production behaviour. The in-CLI legacy plugin keeps serving scans. Reviewers can focus on the plugin code in isolation.

🔬 SPIKE: exploratory work for the multi-ecosystem migration shipped as a draft for early review.

JamesPatrickGill and others added 3 commits June 8, 2026 14:29
@JamesPatrickGill @claude
feat(bundler): add native Gemfile.lock parser and resolver
9152b8d 
Adds pkg/ecosystems/ruby/bundler/ with a native Go parser for Bundler's
Gemfile.lock format and a SCAPlugin that turns it into a Snyk dep graph.

Approach: NATIVE — forced exception to the "delegate to the package
manager's CLI" principle. Every bundler subcommand that emits a dep
graph requires `bundle install` first (mutates the project dir); there
is no install-free CLI path. The lockfile is well-spec'd plain text, so
native parsing in ~150 LoC (ported from @snyk/gemfile) is the only
offline + install-free option.

Coverage:
- All top-level sections: GEM, GIT, PATH, PLATFORMS, DEPENDENCIES,
  BUNDLED WITH, RUBY VERSION.
- Per-spec source metadata (remote/ref/branch/tag/glob).
- DEPENDENCIES `!` marker stripped (e.g. `rspec!` → `rspec`).
- `bundler` itself omitted from the dep tree (matches legacy silent
  drop driven by `if (gemspec)` guard).
- Cycle detection via DFS ancestor chain → CycleError (matches legacy
  422 semantics).
- Group separation hook (BuildOptions.IncludeDev) — wired but no-op
  today because Gemfile.lock doesn't encode groups; the hook is in
  place for a future Gemfile-aware enhancement noted in the migration
  plan as deliberate intent.

Honors --target-file (Gemfile or Gemfile.lock), --all-projects,
--project-name, --exclude-paths. No orchestrator wiring (deliberately
deferred per PR 204 convention).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@JamesPatrickGill @claude
test(bundler): unit tests for parser, depgraph, plugin
64cf699 
33 test cases across parser, dep-graph, and plugin layers:

Parser (parser_test.go, 12 cases):
- Simple GEM block with PLATFORMS/DEPENDENCIES/BUNDLED WITH
- Nested children (indent depth 6) with version constraints stripped
- GIT block + `!` marker stripping on DEPENDENCIES entries
- PATH block (gemspec project)
- Multiple PLATFORMS entries
- RUBY VERSION line
- Empty input / nil reader guard
- DEPENDENCIES variants: bare, pinned, versioned, versioned-and-pinned
- parseSpecLine direct table cases
- parseDependencyLine direct table cases

DepGraph (depgraph_test.go, 9 cases):
- Simple two-dep graph with rubygems pkg-manager metadata
- Transitive resolution with shared ancestry
- Bundler exclusion as both DEPENDENCIES entry and spec child
- Missing-spec graceful skip (matches legacy)
- Cycle detection via DFS ancestor chain → CycleError
- Diamond pattern (shared transitive) — guards against false-cycle bug
- IncludeDev option plumbing
- Nil-lockfile / empty-root guards

Plugin (plugin_test.go, 12 cases):
- Plugin name + identity contract (rootPkg.name = basename(dir))
- No-lockfile returns empty (not an error)
- --target-file accepting Gemfile (resolves to sibling .lock)
- --target-file accepting Gemfile.lock directly
- --target-file rejecting non-gemfile targets
- Custom gemfile names (e.g. rails.2.4.5.gemfile.lock)
- --all-projects discovers nested lockfiles with distinct rootPkg names
- --project-name overrides basename default
- --exclude-paths honored at discovery layer
- Unreadable lockfile handled gracefully
- --dev flag plumbed from options to BuildDepGraphWithOptions
- Nil-options entry tolerated

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@JamesPatrickGill @claude
test(bundler): integration test scaffold with smoke fixtures
e75c5e6 
Adds plugin_integration_test.go (build-tag-gated:
//go:build integration && bundler) plus three smoke fixtures and the
Makefile target to run them.

Fixtures (testdata/acceptance/):
- simple/             — GEM-only lockfile (json + lynx)
- with-git-source/    — two GIT blocks; tests `!` marker stripping for
                        rspec! and nokogiri (= 1.0.0)!
- with-dev-group/     — Gemfile groups :development/:test (json/pry/rspec
                        + transitive); proves the dev-group hook is
                        wired even while group filtering is a no-op

Each fixture commits an expected.json golden. The suite re-runs with
`-update` to refresh: see comment block at the top of the test file.

Critically, the acceptance harness asserts that no `vendor/` or
`.bundle/` directory ever appears in the fixture directory before or
after the run — the principle-#2 ("install free") gate for this
ecosystem. Bundler's install path would create those; the native
parser must never.

Makefile: adds `test-bundler-integration` (mirrors the
test-python-integration / test-gradle-integration convention).

Run: `make test-bundler-integration`
     or `go test -v -tags="integration,bundler" \
                  ./pkg/ecosystems/ruby/bundler/...`

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@JamesPatrickGill JamesPatrickGill added the SPIKE Exploratory / proof-of-concept; not for merge as-is label Jun 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

SPIKE Exploratory / proof-of-concept; not for merge as-is

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@JamesPatrickGill