Skip to content

Conversation

@nickrolfe
Copy link
Contributor

@nickrolfe nickrolfe commented Oct 10, 2025

This adds overlay support to the extractor, along with corresponding QL discard predicates. I have not attempted to add overlay[local] annotations to any libraries, meaning that the 'overlay frontier' is currently at the dbscheme.

Some additional notes:

  • Although the extractor already supported path transformers, it needed a little tweaking to work correctly (and to not be overridden in certain cases).
  • There are a few (pretty rare) cases where analysis of an overlay database will not perfectly match behaviour of full analysis, since we discard anything that is @locatable, but some entities that don't have locations (like types) don't get discarded.
  • There's a bit of a hack to always re-extract cgo-processed files (and, to match that, always to discard entities in such files), since, as far as I can tell, there's no way for the extractor to know which original source file a cgo-processed file came from.

Commit-by-commit review is recommended.

@github-actions github-actions bot added the Go label Oct 10, 2025
@nickrolfe nickrolfe force-pushed the nickrolfe/go-extractor-overlay branch from ffc7fee to 199b8fc Compare October 23, 2025 15:03
@nickrolfe nickrolfe marked this pull request as ready for review October 24, 2025 10:19
@nickrolfe nickrolfe requested a review from a team as a code owner October 24, 2025 10:19
Copilot AI review requested due to automatic review settings October 24, 2025 10:19
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds foundational support for database overlay functionality in the Go extractor, enabling incremental analysis by reusing a base database and overlaying only the changes from modified files.

Key changes:

  • Introduces new database schema relations (databaseMetadata and overlayChangedFiles) to support overlay metadata
  • Adds logic to skip extraction of unchanged files when building an overlay database
  • Updates path handling to use transformed paths consistently across extraction and trap file generation

Reviewed Changes

Copilot reviewed 23 out of 23 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
go/ql/lib/upgrades/b3da71c3ac204b557c86e9d9c26012360bdbdccb/upgrade.properties Defines upgrade metadata for adding overlay support relations
go/ql/lib/upgrades/b3da71c3ac204b557c86e9d9c26012360bdbdccb/old.dbscheme Baseline schema snapshot before overlay changes
go/ql/lib/upgrades/b3da71c3ac204b557c86e9d9c26012360bdbdccb/go.dbscheme Updated schema with overlay support relations
go/ql/lib/semmle/go/Overlay.qll Defines entity discard predicates for overlay analysis
go/ql/lib/semmle/go/Locations.qll Imports overlay module for location handling
go/ql/lib/qlpack.yml Enables overlay evaluation compilation
go/ql/lib/go.dbscheme Adds databaseMetadata and overlayChangedFiles relations
go/extractor/util/overlays.go Implements overlay detection and changed file handling
go/extractor/util/BUILD.bazel Adds overlays.go to build sources
go/extractor/trap/labels.go Updates file label generation to use transformed paths
go/extractor/srcarchive/srcarchive.go Refactors path transformer initialization
go/extractor/srcarchive/projectlayout.go Adds environment-based project layout loader and exports fields
go/extractor/srcarchive/BUILD.bazel Adds util dependency
go/extractor/gomodextractor.go Skips extraction of unchanged go.mod files in overlay mode
go/extractor/extractor.go Integrates overlay change tracking and skips unchanged file extraction
go/extractor/dbscheme/tables.go Adds overlay relations to schema definition
go/extractor/cli/go-extractor/go-extractor.go Adds --source-root argument parsing
go/extractor/cli/go-autobuilder/go-autobuilder.go Passes source root to extractor and writes overlay metadata
go/extractor/cli/go-autobuilder/BUILD.bazel Adds srcarchive dependency
go/downgrades/b1341734d6870b105e5c9d168ce7dec25d7f72d0/upgrade.properties Defines downgrade operation for overlay relations
go/downgrades/b1341734d6870b105e5c9d168ce7dec25d7f72d0/old.dbscheme Schema with overlay relations for downgrade reference
go/downgrades/b1341734d6870b105e5c9d168ce7dec25d7f72d0/go.dbscheme Target schema without overlay relations
go/codeql-extractor.yml Declares overlay support version

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

if err != nil {
return nil, err
}
pathTransformer, err = LoadProjectLayout(ptf)
Copy link

Copilot AI Oct 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pathTransformer variable being assigned here is a package-level variable, but in the context of this function, a local variable should be used instead. The function returns a *ProjectLayout, so the assignment should be to a local variable that is then returned, not to the package-level pathTransformer.

Suggested change
pathTransformer, err = LoadProjectLayout(ptf)
pathTransformer, err := LoadProjectLayout(ptf)

Copilot uses AI. Check for mistakes.
// extract a file if it's not in the package directory.
if extraction.OverlayChanges != nil &&
!extraction.OverlayChanges[path] &&
strings.HasPrefix(path+string(filepath.Separator), pkg.Dir) {
Copy link

Copilot AI Oct 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The path prefix check is incorrect. It should use pkg.Dir+string(filepath.Separator) as the prefix to check, not append the separator to path. The current logic checks if path concatenated with a separator starts with pkg.Dir, which is backwards.

Suggested change
strings.HasPrefix(path+string(filepath.Separator), pkg.Dir) {
strings.HasPrefix(path, pkg.Dir+string(filepath.Separator)) {

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants