-
Couldn't load subscription status.
- Fork 1.8k
Go: basic overlay support #20623
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Go: basic overlay support #20623
Conversation
When in overlay mode, extractFile will exit early if the file isn't in the list of files that changed since the base was extracted.
This ensures the extractor can resolve the relative paths for files changed in the overlay.
ffc7fee to
199b8fc
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR adds foundational support for database overlay functionality in the Go extractor, enabling incremental analysis by reusing a base database and overlaying only the changes from modified files.
Key changes:
- Introduces new database schema relations (
databaseMetadataandoverlayChangedFiles) to support overlay metadata - Adds logic to skip extraction of unchanged files when building an overlay database
- Updates path handling to use transformed paths consistently across extraction and trap file generation
Reviewed Changes
Copilot reviewed 23 out of 23 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| go/ql/lib/upgrades/b3da71c3ac204b557c86e9d9c26012360bdbdccb/upgrade.properties | Defines upgrade metadata for adding overlay support relations |
| go/ql/lib/upgrades/b3da71c3ac204b557c86e9d9c26012360bdbdccb/old.dbscheme | Baseline schema snapshot before overlay changes |
| go/ql/lib/upgrades/b3da71c3ac204b557c86e9d9c26012360bdbdccb/go.dbscheme | Updated schema with overlay support relations |
| go/ql/lib/semmle/go/Overlay.qll | Defines entity discard predicates for overlay analysis |
| go/ql/lib/semmle/go/Locations.qll | Imports overlay module for location handling |
| go/ql/lib/qlpack.yml | Enables overlay evaluation compilation |
| go/ql/lib/go.dbscheme | Adds databaseMetadata and overlayChangedFiles relations |
| go/extractor/util/overlays.go | Implements overlay detection and changed file handling |
| go/extractor/util/BUILD.bazel | Adds overlays.go to build sources |
| go/extractor/trap/labels.go | Updates file label generation to use transformed paths |
| go/extractor/srcarchive/srcarchive.go | Refactors path transformer initialization |
| go/extractor/srcarchive/projectlayout.go | Adds environment-based project layout loader and exports fields |
| go/extractor/srcarchive/BUILD.bazel | Adds util dependency |
| go/extractor/gomodextractor.go | Skips extraction of unchanged go.mod files in overlay mode |
| go/extractor/extractor.go | Integrates overlay change tracking and skips unchanged file extraction |
| go/extractor/dbscheme/tables.go | Adds overlay relations to schema definition |
| go/extractor/cli/go-extractor/go-extractor.go | Adds --source-root argument parsing |
| go/extractor/cli/go-autobuilder/go-autobuilder.go | Passes source root to extractor and writes overlay metadata |
| go/extractor/cli/go-autobuilder/BUILD.bazel | Adds srcarchive dependency |
| go/downgrades/b1341734d6870b105e5c9d168ce7dec25d7f72d0/upgrade.properties | Defines downgrade operation for overlay relations |
| go/downgrades/b1341734d6870b105e5c9d168ce7dec25d7f72d0/old.dbscheme | Schema with overlay relations for downgrade reference |
| go/downgrades/b1341734d6870b105e5c9d168ce7dec25d7f72d0/go.dbscheme | Target schema without overlay relations |
| go/codeql-extractor.yml | Declares overlay support version |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
| if err != nil { | ||
| return nil, err | ||
| } | ||
| pathTransformer, err = LoadProjectLayout(ptf) |
Copilot
AI
Oct 24, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The pathTransformer variable being assigned here is a package-level variable, but in the context of this function, a local variable should be used instead. The function returns a *ProjectLayout, so the assignment should be to a local variable that is then returned, not to the package-level pathTransformer.
| pathTransformer, err = LoadProjectLayout(ptf) | |
| pathTransformer, err := LoadProjectLayout(ptf) |
| // extract a file if it's not in the package directory. | ||
| if extraction.OverlayChanges != nil && | ||
| !extraction.OverlayChanges[path] && | ||
| strings.HasPrefix(path+string(filepath.Separator), pkg.Dir) { |
Copilot
AI
Oct 24, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The path prefix check is incorrect. It should use pkg.Dir+string(filepath.Separator) as the prefix to check, not append the separator to path. The current logic checks if path concatenated with a separator starts with pkg.Dir, which is backwards.
| strings.HasPrefix(path+string(filepath.Separator), pkg.Dir) { | |
| strings.HasPrefix(path, pkg.Dir+string(filepath.Separator)) { |
This adds overlay support to the extractor, along with corresponding QL discard predicates. I have not attempted to add
overlay[local]annotations to any libraries, meaning that the 'overlay frontier' is currently at the dbscheme.Some additional notes:
@locatable, but some entities that don't have locations (like types) don't get discarded.Commit-by-commit review is recommended.