feat: implement hybrid cache architecture #1246

fabiovincenzi · 2025-10-13T11:28:03Z

Summary

Two-tier cache system for faster git operations: persistent bare repos + ephemeral working copies.

Architecture

.remote/cache/ - Shared bare repositories (persistent)
.remote/work/<push-id>/ - Per-push working copies (temporary)

Configuration

Add to proxy.config.json:

{
  "cache": {
    "maxSizeGB": 2,
    "maxRepositories": 50,
    "cacheDir": "./.remote/cache"
  }
}

Security

Each push gets unique working copy (.remote/work//)
Bare cache contains only git objects (no user files, no credentials)
sanitizeRepositoryConfig() immediately removes credentials from git config
Working copies deleted after push completes

netlify · 2025-10-13T11:28:23Z

✅ Deploy Preview for endearing-brigadeiros-63f9d0 canceled.

Name	Link
🔨 Latest commit	`133e5e6`
🔍 Latest deploy log	https://app.netlify.com/projects/endearing-brigadeiros-63f9d0/deploys/68ff7d335b586300080bbc85

codecov · 2025-10-13T11:32:06Z

Codecov Report

❌ Patch coverage is 83.24873% with 33 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.66%. Comparing base (6f56f15) to head (627137b).

Files with missing lines	Patch %	Lines
src/proxy/processors/push-action/clearBareClone.ts	37.50%	9 Missing and 1 partial ⚠️
src/proxy/processors/push-action/git-operations.ts	81.63%	6 Missing and 3 partials ⚠️
src/proxy/processors/push-action/pullRemote.ts	85.41%	7 Missing ⚠️
src/proxy/processors/push-action/cache-manager.ts	90.62%	6 Missing ⚠️
src/config/index.ts	83.33%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1246      +/-   ##
==========================================
+ Coverage   82.64%   82.66%   +0.02%     
==========================================
  Files          70       73       +3     
  Lines        3007     3185     +178     
  Branches      501      537      +36     
==========================================
+ Hits         2485     2633     +148     
- Misses        419      444      +25     
- Partials      103      108       +5

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Signed-off-by: Fabio Vincenzi <93596376+fabiovincenzi@users.noreply.github.com>

…zi/git-proxy into feature/hybrid-cache

06kellyjac

I'm looking forward to faster pulls!

Caches are a common place for logic or security problems so we may want added scrutiny and testing for this PR to be extra safe :)

Added some thoughts, not ran this locally just yet.

06kellyjac · 2025-10-24T13:46:56Z

src/proxy/processors/push-action/clearBareClone.ts

+  if (process.env.NODE_ENV === 'test') {
+    // TEST: Full cleanup (bare cache + all working copies)
+    try {
+      if (fs.existsSync('./.remote')) {
+        fs.rmSync('./.remote', { recursive: true, force: true });
+        step.log('Test environment: Full .remote directory cleaned');
+      } else {
+        step.log('Test environment: .remote directory already clean');
+      }
+    } catch (err) {
+      step.log(`Warning: Could not clean .remote directory: ${err}`);
+    }
+  } else {


I feel like a safer option that ensures what we're testing more closely matches prod would be to either do some separate cleanup as necessary in the tests, or adjust config to have a much lower max (or even 0) then we can see if CacheManager itself is doing the cleanup.

src/proxy/processors/push-action/git-operations.ts

src/proxy/processors/push-action/pullRemote.ts

src/config/index.ts

06kellyjac · 2025-10-24T14:01:07Z

src/proxy/processors/push-action/cache-manager.ts

+    // Sort repositories by last accessed (oldest first for removal)
+    const reposToEvaluate = [...stats.repositories].sort(
+      (a, b) => a.lastAccessed.getTime() - b.lastAccessed.getTime(),
+    );


I can't remember if we can use toSorted yet. If we still cant a comment to use toSorted in the future would help to find this later :)

I think it should be available from node 20 so yes I'll use it

src/proxy/processors/push-action/cache-manager.ts

06kellyjac · 2025-10-24T14:12:33Z

src/proxy/processors/push-action/cache-manager.ts

+      return 0;
+    }
+
+    return Math.round(totalBytes / (1024 * 1024)); // Convert to MB


Suggested change

return Math.round(totalBytes / (1024 * 1024)); // Convert to MB

return Math.ceil(totalBytes / (1024 * 1024)); // Convert to MB

rounding down would hide some amount of KB up to 511KB. Not the end of the world but safest to always round up rather than be shortchanged.

There's no conversion anymore since we are using bytes everywhere

jescalada

Just a code review for now - I'll be testing the code later!

I was wondering if you could do some profiling to verify/document the speed improvements with this PR. Would be nice to try it out with massive repos (for example https://github.com/backstage/backstage which is >10GB) to see the difference.

I'll try to do some profiling soon as well 🙂

jescalada · 2025-10-27T09:33:46Z

src/proxy/processors/push-action/cache-manager.ts

+}
+
+export class CacheManager {
+  private cacheDir: string;


Is this cacheDir related to the cachedir library in package.json or the cacheDir used in the ConfigLoader? If not, we might want to pick a different name to disambiguate them.

jescalada · 2025-10-27T09:55:02Z

src/proxy/processors/push-action/cache-manager.ts

I'm wondering what would happen if two pulls happen simultaneously and execute touchRepository or enforceLimits at the same time - wouldn't this cause some kind of race condition? Especially with cache statistics in the mix, we wouldn't be able to accurately pick up the latest accessed repository.

jescalada · 2025-10-27T10:50:16Z

src/proxy/processors/push-action/cache-manager.ts

+    let freedMB = 0;
+
+    // Sort repositories by last accessed (oldest first for removal)
+    const reposToEvaluate = [...stats.repositories].sort(


Is there a reason to sort the repositories twice? The first sort happens inside the getCacheStats() call right above.

jescalada · 2025-10-27T10:56:36Z

src/proxy/processors/push-action/cache-manager.ts

+  /**
+   * Calculate directory size in MB
+   */
+  private getDirectorySize(dirPath: string): number {


Should the error clauses here be logging something instead of failing silently? Such as when there's an error on calculateSize(dirPath).

jescalada · 2025-10-27T11:15:37Z

src/proxy/processors/push-action/pullRemote.ts

Would be nice to have a short README on how the pullRemote works and what the hybrid cache architecture is doing. This'll make things much clearer for future development and finding potential flaws in our logic 🙂

06kellyjac · 2025-10-27T11:50:35Z

why on earth is backstage's repo 10GB? 😭
Even nixpkgs is only 6.1GB ATM on my machine

jescalada · 2025-10-27T12:50:24Z

@06kellyjac It might be because of the tons of plugins that come with the repo (many of which have redundant components). Pulling with git clone --filter=blob:none https://github.com/backstage/backstage.git seems to make things a lot more speedy, however I'm not sure if we want to do things that way in the pullRemote. Perhaps a config option to filter out the BLOBs could help with these massive repos?

fabiovincenzi added 4 commits October 10, 2025 16:18

feat: implement hybrid cache architecture for repository cloning

956f388

feat: add configurable cache limits via proxy.config.json

cc49057

test: update clearBareClone tests for hybrid cache structure

d6413ce

chore: fix metric logging

cf76665

github-actions bot added the feature label Oct 13, 2025

Merge branch 'main' into feature/hybrid-cache

e998d07

fabiovincenzi mentioned this pull request Oct 13, 2025

Improve push flexibility and pull times #1207

Open

Merge branch 'main' into feature/hybrid-cache

734621c

fabiovincenzi force-pushed the feature/hybrid-cache branch from 12b56bd to 734621c Compare October 20, 2025 07:42

fabiovincenzi and others added 7 commits October 20, 2025 09:44

Merge branch 'main' into feature/hybrid-cache

452eb18

Signed-off-by: Fabio Vincenzi <93596376+fabiovincenzi@users.noreply.github.com>

feat: implement git-operations module with native git commands

168d9b0

refactor: update pullRemote to use git-operations module

08116ba

test: add comprehensive hybrid cache integration tests

992c862

chore: remove unused isomorphic-git dependency

d5e1b5b

Merge branch 'feature/hybrid-cache' of https://github.com/fabiovincen…

6d5f886

…zi/git-proxy into feature/hybrid-cache

Merge branch 'main' into feature/hybrid-cache

6bc0ddc

fabiovincenzi marked this pull request as ready for review October 22, 2025 14:13

fabiovincenzi requested review from 06kellyjac, jescalada and kriswest October 22, 2025 14:18

06kellyjac reviewed Oct 24, 2025

View reviewed changes

fabiovincenzi and others added 4 commits October 27, 2025 11:00

Merge branch 'main' into feature/hybrid-cache

627137b

chore: remove redundant cache config fallback

235e152

refactor: use bytes internally in CacheManager for consistency

c762b5e

chore: use toSorted()

ab28d78

jescalada reviewed Oct 27, 2025

View reviewed changes

fabiovincenzi added 5 commits October 27, 2025 14:38

refactor: remove test-only cleanup, rely on CacheManager limits

88bbce8

refactor: use spawnSync instead of execSync

1737bfd

test: increase timeout for git clone tests in ConfigLoader

d35d109

perf: use performance.now() instead of Date.now()

0a54773

refactor: use cache paths from configuration

133e5e6

	return Math.round(totalBytes / (1024 * 1024)); // Convert to MB
	return Math.ceil(totalBytes / (1024 * 1024)); // Convert to MB

Uh oh!

feat: implement hybrid cache architecture #1246

Are you sure you want to change the base?

feat: implement hybrid cache architecture #1246

Uh oh!

Conversation

fabiovincenzi commented Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Architecture

Configuration

Security

Uh oh!

netlify bot commented Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for endearing-brigadeiros-63f9d0 canceled.

Uh oh!

codecov bot commented Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

06kellyjac left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jescalada left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

06kellyjac commented Oct 27, 2025

Uh oh!

jescalada commented Oct 27, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

fabiovincenzi commented Oct 13, 2025 •

edited

Loading

netlify bot commented Oct 13, 2025 •

edited

Loading

codecov bot commented Oct 13, 2025 •

edited

Loading