From f1cc086ae844cb2fef28bfdf690c02813e6b7f79 Mon Sep 17 00:00:00 2001 From: kylegalbraith <1869717+kylegalbraith@users.noreply.github.com> Date: Sat, 11 Oct 2025 13:15:25 +0000 Subject: [PATCH] Update docs content from https://github.com/depot/app --- content/agents/claude-code/quickstart.mdx | 34 +-- content/cache/reference/bazel.mdx | 42 +++- content/cache/reference/gocache.mdx | 45 +++- content/cache/reference/gradle.mdx | 57 ++++- content/cache/reference/maven.mdx | 85 +++++++- content/cache/reference/moonrepo.mdx | 52 ++++- content/cache/reference/pants.mdx | 53 ++++- content/cache/reference/sccache.mdx | 41 +++- content/cache/reference/turbo.mdx | 18 +- .../container-builds/build-parallelism.mdx | 206 ++++++++++++++++++ .../how-to-guides/private-registries.mdx | 2 +- content/container-builds/quickstart.mdx | 32 +-- .../access-private-resources.mdx | 114 ++++++++++ content/github-actions/quickstart.mdx | 6 +- 14 files changed, 729 insertions(+), 58 deletions(-) create mode 100644 content/container-builds/build-parallelism.mdx create mode 100644 content/github-actions/how-to-guides/access-private-resources.mdx diff --git a/content/agents/claude-code/quickstart.mdx b/content/agents/claude-code/quickstart.mdx index 2571c69..b85ad4e 100644 --- a/content/agents/claude-code/quickstart.mdx +++ b/content/agents/claude-code/quickstart.mdx @@ -9,31 +9,33 @@ import {DocsCTA} from '~/components/blog/CTA' Run Claude Code agents in Depot's remote agent sandboxes: a secure, isolated environment in the cloud, where you can launch, resume, and share coding sessions. -## Install the Depot CLI +## Prerequisites -You'll need the [Depot CLI](/docs/cli/reference) on your machine to configure and launch remote sandboxes. +You'll need a [Depot account](https://depot.dev/sign-up). -### macOS +## Install the Depot CLI -Install the Depot CLI with Homebrew: +Install the [Depot CLI](/docs/cli/reference) on your machine to configure and launch remote sandboxes. -```shell -brew install depot/tap/depot -``` +- **macOS** -### Linux + Install the Depot CLI with Homebrew: -Install the Depot CLI with the installation script. + ```shell + brew install depot/tap/depot + ``` -Install the latest version: +- **Linux** -```shell -curl -L https://depot.dev/install-cli.sh | sh -``` + Install the Depot CLI with the installation script: + + ```shell + curl -L https://depot.dev/install-cli.sh | sh + ``` -### All platforms +- **All platforms** -Download the binary file for your platform from the [Depot CLI releases page](https://github.com/depot/cli/releases) in GitHub. + Download the binary file for your platform from the [Depot CLI releases page](https://github.com/depot/cli/releases) in GitHub. ## Get and set your Anthropic credentials @@ -87,7 +89,7 @@ To grant remote agent sandboxes access to clone and push changes to your private ### Grant access outside of GitHub -If you don't want to use the Depot Code app, you can set your Git credentials as secrets in your Depot organization allow changes to your private repositories. The value of `GIT_CREDENTIALS` must be one of the following: +If you don't want to use the Depot Code app, you can set your Git credentials as secrets in your Depot organization to allow changes to your private repositories. The value of `GIT_CREDENTIALS` must be one of the following: - A token, such as a personal access token. Depot uses `x-token` as the username and the token you specify as the password. - A user name and password in the format: username@password. diff --git a/content/cache/reference/bazel.mdx b/content/cache/reference/bazel.mdx index 29a9e7d..c8039bd 100644 --- a/content/cache/reference/bazel.mdx +++ b/content/cache/reference/bazel.mdx @@ -33,6 +33,44 @@ If you are a member of multiple organizations, and you are authenticating with a build --remote_header=x-depot-org=DEPOT_ORG_ID ``` -## Using Depot Cache with Bazel - Once Bazel is configured to use Depot Cache, you can then run your builds as you normally would. Bazel will automatically communicate with Depot Cache to fetch and reuse any stored build artifacts from your previous builds. + +### Using Depot Cache with Bazel in `depot/build-push-action` + +When using `depot/build-push-action` to build Docker images that contain Bazel workspaces, your build needs access to Bazel's remote cache credentials to benefit from caching. + +These credentials are not automatically available inside your Docker build environment. Unlike builds running directly on Depot-managed GitHub Actions runners (which have automatic access to Depot Cache environment variables), containerized builds execute in isolated VMs that require explicit configuration. + +Follow these steps to securely pass your Bazel credentials into your Docker build: + +1. Store the Depot token in a GitHub Secret named `DEPOT_TOKEN`. + +2. Configure your GitHub Action to pass secrets to the container build: + +```yaml +- name: Build and push + uses: depot/build-push-action@v1 + with: + context: . + file: ./Dockerfile + push: true + tags: your-image:tag + secrets: | + "DEPOT_TOKEN=${{ secrets.DEPOT_TOKEN }}" +``` + +3. Update your Dockerfile to mount the secrets and configure Bazel: + +```dockerfile +# syntax=docker/dockerfile:1 + +# ... other Dockerfile instructions + +# Create .bazelrc with cache configuration +RUN --mount=type=secret,id=DEPOT_TOKEN,env=DEPOT_TOKEN \ + echo "build --remote_cache=https://cache.depot.dev" >> ~/.bazelrc && \ + echo "build --remote_header=authorization=${DEPOT_TOKEN}" >> ~/.bazelrc && \ + bazel build +``` + +Adding `# syntax=docker/dockerfile:1` as the first line of your Dockerfile enables mounting secrets as environment variables. diff --git a/content/cache/reference/gocache.mdx b/content/cache/reference/gocache.mdx index 5dcc6b4..eaed62e 100644 --- a/content/cache/reference/gocache.mdx +++ b/content/cache/reference/gocache.mdx @@ -42,6 +42,47 @@ To set verbose output, add the --verbose option: export GOCACHEPROG='depot gocache --verbose' ``` -## Using Depot Cache with Go - Once Go is configured to use Depot Cache, you can then run your builds as you normally would. Go will automatically communicate with `GOCACHEPROG` to fetch from Depot Cache and reuse any stored build artifacts from your previous builds. + +### Using Depot Cache with Go in `depot/build-push-action` + +When using `depot/build-push-action` to build Docker images that contain Go projects, your build needs access to Go's remote cache credentials to benefit from caching. + +These credentials are not automatically available inside your Docker build environment. Unlike builds running directly on Depot-managed GitHub Actions runners (which have automatic access to Depot Cache environment variables), containerized builds execute in isolated VMs that require explicit configuration. + +Follow these steps to securely pass your Go cache credentials into your Docker build: + +1. Store the Depot token in a GitHub Secret named `DEPOT_TOKEN`. + +2. Configure your GitHub Action to pass secrets to the container build: + +```yaml +- name: Build and push + uses: depot/build-push-action@v1 + with: + context: . + file: ./Dockerfile + push: true + tags: your-image:tag + secrets: | + "DEPOT_TOKEN=${{ secrets.DEPOT_TOKEN }}" +``` + +3. Update your Dockerfile to install the Depot CLI and configure Go cache: + +```dockerfile +# syntax=docker/dockerfile:1 + +# ... other Dockerfile instructions + +# Install Depot CLI +RUN curl -L https://depot.dev/install-cli.sh | sh + +# Mount secret and set GOCACHEPROG +RUN --mount=type=secret,id=DEPOT_TOKEN,env=DEPOT_TOKEN \ + PATH="/root/.depot/bin:$PATH" \ + GOCACHEPROG="depot gocache" \ + go build -v ./ +``` + +Adding `# syntax=docker/dockerfile:1` as the first line of your Dockerfile enables mounting secrets as environment variables. diff --git a/content/cache/reference/gradle.mdx b/content/cache/reference/gradle.mdx index 68ba841..591d092 100644 --- a/content/cache/reference/gradle.mdx +++ b/content/cache/reference/gradle.mdx @@ -58,6 +58,59 @@ buildCache { } ``` -## Using Depot Cache with Gradle - Once Gradle is configured to use Depot Cache, you can then run your builds as you normally would. Gradle will automatically communicate with Depot Cache to fetch and reuse any stored build artifacts from your previous builds. + +### Using Depot Cache with Gradle in `depot/build-push-action` + +When using `depot/build-push-action` to build Docker images that contain Gradle projects, your build needs access to Gradle's remote cache credentials to benefit from caching. + +These credentials are not automatically available inside your Docker build environment. Unlike builds running directly on Depot-managed GitHub Actions runners (which have automatic access to Depot Cache environment variables), containerized builds execute in isolated VMs that require explicit configuration. + +Follow these steps to securely pass your Gradle credentials into your Docker build: + +1. Store the Depot token in a GitHub Secret named `DEPOT_TOKEN`. + +2. Update your `settings.gradle` to read the Depot token from an environment variable: + +```groovy +buildCache { + remote(HttpBuildCache) { + url = 'https://cache.depot.dev' + enabled = true + push = true + credentials { + username = '' + password = System.getenv('DEPOT_TOKEN') + } + } +} +``` + +3. Configure your GitHub Action to pass secrets to the container build: + +```yaml +- name: Build and push + uses: depot/build-push-action@v1 + with: + context: . + file: ./Dockerfile + push: true + tags: your-image:tag + secrets: | + "DEPOT_TOKEN=${{ secrets.DEPOT_TOKEN }}" +``` + +4. Update your Dockerfile to mount the secret and run the build: + +```dockerfile +# syntax=docker/dockerfile:1 + +# ... other Dockerfile instructions + +# Copy settings.gradle and run build with mounted secret +COPY settings.gradle . +RUN --mount=type=secret,id=DEPOT_TOKEN,env=DEPOT_TOKEN \ + ./gradlew build +``` + +Adding `# syntax=docker/dockerfile:1` as the first line of your Dockerfile enables mounting secrets as environment variables. diff --git a/content/cache/reference/maven.mdx b/content/cache/reference/maven.mdx index e5addc3..9ed734e 100644 --- a/content/cache/reference/maven.mdx +++ b/content/cache/reference/maven.mdx @@ -77,6 +77,87 @@ To manually configure Maven to use Depot Cache, you will need to configure remot **Note: Maven support currently only supports Depot Organization API tokens, not user tokens.** -## Using Depot Cache with Maven - Once Maven is configured to use Depot Cache, you can run your builds as usual. Maven will automatically communicate with Depot Cache to fetch and reuse any stored build artifacts from your previous builds. + +### Using Depot Cache with Maven in `depot/build-push-action` + +When using `depot/build-push-action` to build Docker images that contain Maven projects, your build needs access to Maven's remote cache credentials to benefit from caching. + +These credentials are not automatically available inside your Docker build environment. Unlike builds running directly on Depot-managed GitHub Actions runners (which have automatic access to Depot Cache environment variables), containerized builds execute in isolated VMs that require explicit configuration. + +Follow these steps to securely pass your Maven credentials into your Docker build: + +1. Store the Depot token in a GitHub Secret named `DEPOT_TOKEN`. + +2. Configure Maven Build Cache extension in `.mvn/maven-build-cache-config.xml`: + +```xml + + + true + SHA-256 + true + + https://cache.depot.dev + + + + +``` + +3. Update your `settings.xml` to read the Depot token from an environment variable. Create or update `.m2/settings.xml`: + +```xml + + + + + depot-cache + + + + Authorization + Bearer ${env.DEPOT_TOKEN} + + + + + + +``` + +4. Configure your GitHub Action to pass secrets to the container build: + +```yaml +- name: Build and push + uses: depot/build-push-action@v1 + with: + context: . + file: ./Dockerfile + push: true + tags: your-image:tag + secrets: | + "DEPOT_TOKEN=${{ secrets.DEPOT_TOKEN }}" +``` + +5. Update your Dockerfile to copy configuration files and run the build with mounted secret: + +```dockerfile +# syntax=docker/dockerfile:1 + +# ... other Dockerfile instructions + +# Copy Maven configuration files +COPY .mvn/maven-build-cache-config.xml .mvn/maven-build-cache-config.xml +COPY .m2/settings.xml /root/.m2/settings.xml + +# Run build with mounted secret +RUN --mount=type=secret,id=DEPOT_TOKEN,env=DEPOT_TOKEN \ + mvn clean install +``` + +Adding `# syntax=docker/dockerfile:1` as the first line of your Dockerfile enables mounting secrets as environment variables. diff --git a/content/cache/reference/moonrepo.mdx b/content/cache/reference/moonrepo.mdx index 4a5e88b..e13f150 100644 --- a/content/cache/reference/moonrepo.mdx +++ b/content/cache/reference/moonrepo.mdx @@ -34,6 +34,54 @@ unstable_remote: See [moonrepo's remote cache documentation](https://moonrepo.dev/docs/guides/remote-cache#cloud-hosted-depot) for more details. -## Using Depot Cache with moonrepo - Once moonrepo is configured to use Depot Cache, you can then run your builds as you normally would. moonrepo will automatically communicate with Depot Cache to fetch and reuse any stored build artifacts from your previous builds. + +### Using Depot Cache with moonrepo in `depot/build-push-action` + +When using `depot/build-push-action` to build Docker images that contain moonrepo workspaces, your build needs access to moonrepo's remote cache credentials to benefit from caching. + +These credentials are not automatically available inside your Docker build environment. Unlike builds running directly on Depot-managed GitHub Actions runners (which have automatic access to Depot Cache environment variables), containerized builds execute in isolated VMs that require explicit configuration. + +Follow these steps to securely pass your moonrepo credentials into your Docker build: + +1. Store the Depot token in a GitHub Secret named `DEPOT_TOKEN`. + +2. Configure moonrepo to read the Depot token from an environment variable in `.moon/workspace.yml`: + +```yaml +unstable_remote: + host: 'grpcs://cache.depot.dev' + auth: + token: 'DEPOT_TOKEN' +``` + +3. Configure your GitHub Action to pass secrets to the container build: + +```yaml +- name: Build and push + uses: depot/build-push-action@v1 + with: + context: . + file: ./Dockerfile + push: true + tags: your-image:tag + secrets: | + "DEPOT_TOKEN=${{ secrets.DEPOT_TOKEN }}" +``` + +4. Update your Dockerfile to copy the workspace configuration and run the build with mounted secret: + +```dockerfile +# syntax=docker/dockerfile:1 + +# ... other Dockerfile instructions + +# Copy moonrepo workspace configuration +COPY .moon/workspace.yml .moon/workspace.yml + +# Mount secret as environment variable and run build +RUN --mount=type=secret,id=DEPOT_TOKEN,env=DEPOT_TOKEN \ + moon run build +``` + +Adding `# syntax=docker/dockerfile:1` as the first line of your Dockerfile enables mounting secrets as environment variables. diff --git a/content/cache/reference/pants.mdx b/content/cache/reference/pants.mdx index d71619f..755a0c7 100644 --- a/content/cache/reference/pants.mdx +++ b/content/cache/reference/pants.mdx @@ -41,6 +41,55 @@ If you are a member of multiple organizations, and you are authenticating with a remote_store_headers = { "x-depot-org" = "DEPOT_ORG_ID" } ``` -## Using Depot Cache with Pants - Once Pants is configured to use Depot Cache, you can then run your builds as you normally would. Pants will automatically communicate with Depot Cache to fetch and reuse any stored build artifacts from your previous builds. + +### Using Depot Cache with Pants in `depot/build-push-action` + +When using `depot/build-push-action` to build Docker images that contain Pants projects, your build needs access to Pants' remote cache credentials to benefit from caching. + +These credentials are not automatically available inside your Docker build environment. Unlike builds running directly on Depot-managed GitHub Actions runners (which have automatic access to Depot Cache environment variables), containerized builds execute in isolated VMs that require explicit configuration. + +Follow these steps to securely pass your Pants credentials into your Docker build: + +1. Store the Depot token in a GitHub Secret named `DEPOT_TOKEN`. + +2. Update your `pants.toml` to read the Depot token from an environment variable: + +```toml +[GLOBAL] +remote_cache_read = true +remote_cache_write = true +remote_store_address = "grpcs://cache.depot.dev" + +[GLOBAL.remote_store_headers] +Authorization = "%(env.DEPOT_TOKEN)s" +``` + +3. Configure your GitHub Action to pass secrets to the container build: + +```yaml +- name: Build and push + uses: depot/build-push-action@v1 + with: + context: . + file: ./Dockerfile + push: true + tags: your-image:tag + secrets: | + "DEPOT_TOKEN=${{ secrets.DEPOT_TOKEN }}" +``` + +4. Update your Dockerfile to mount the secret and run the build: + +```dockerfile +# syntax=docker/dockerfile:1 + +# ... other Dockerfile instructions + +# Copy pants.toml and run build with mounted secret +COPY pants.toml . +RUN --mount=type=secret,id=DEPOT_TOKEN,env=DEPOT_TOKEN \ + pants package :: +``` + +Adding `# syntax=docker/dockerfile:1` as the first line of your Dockerfile enables mounting secrets as environment variables. diff --git a/content/cache/reference/sccache.mdx b/content/cache/reference/sccache.mdx index 2045bab..e9fd0ea 100644 --- a/content/cache/reference/sccache.mdx +++ b/content/cache/reference/sccache.mdx @@ -35,6 +35,43 @@ export SCCACHE_WEBDAV_USERNAME=DEPOT_ORG_ID export SCCACHE_WEBDAV_PASSWORD=DEPOT_TOKEN ``` -## Using Depot Cache with sccache - Once sccache is configured to use Depot Cache, you can then run your builds as you normally would. sccache will automatically communicate with Depot Cache to fetch and reuse any stored build artifacts from your previous builds. + +### Using Depot Cache with sccache in `depot/build-push-action` + +When using `depot/build-push-action` to build Docker images that contain Rust projects with sccache, your build needs access to sccache's remote cache credentials to benefit from caching. + +These credentials are not automatically available inside your Docker build environment. Unlike builds running directly on Depot-managed GitHub Actions runners (which have automatic access to Depot Cache environment variables), containerized builds execute in isolated VMs that require explicit configuration. + +Follow these steps to securely pass your sccache credentials into your Docker build: + +1. Store the Depot token in a GitHub Secret named `DEPOT_TOKEN`. + +2. Configure your GitHub Action to pass secrets to the container build: + +```yaml +- name: Build and push + uses: depot/build-push-action@v1 + with: + context: . + file: ./Dockerfile + push: true + tags: your-image:tag + secrets: | + "DEPOT_TOKEN=${{ secrets.DEPOT_TOKEN }}" +``` + +3. Update your Dockerfile to mount the secrets as environment variables: + +```dockerfile +# syntax=docker/dockerfile:1 + +# ... other Dockerfile instructions + +# Mount secrets with IDs matching the environment variable names +RUN --mount=type=secret,id=DEPOT_TOKEN,env=SCCACHE_WEBDAV_TOKEN \ + SCCACHE_WEBDAV_ENDPOINT=https://cache.depot.dev sccache --start-server && \ + cargo build --release +``` + +Adding `# syntax=docker/dockerfile:1` as the first line of your Dockerfile enables mounting secrets as environment variables. diff --git a/content/cache/reference/turbo.mdx b/content/cache/reference/turbo.mdx index 86155d6..4256621 100644 --- a/content/cache/reference/turbo.mdx +++ b/content/cache/reference/turbo.mdx @@ -28,20 +28,15 @@ export TURBO_TOKEN=DEPOT_TOKEN export TURBO_TEAM=DEPOT_ORG_ID ``` -## Using Depot Cache with Turborepo - Once Turborepo is configured to use Depot Cache, you can then run your builds as you normally would. Turborepo will automatically communicate with Depot Cache to fetch and reuse any stored build artifacts from your previous builds. -### Using Depot Cache with Turborepo in container builds - -Container builds with `depot/build-push-action` don't automatically connect to Depot Cache. Unlike Depot GitHub Actions runners, container builds run on separate VMs and require manual configuration. +### Using Depot Cache with Turborepo in `depot/build-push-action` -You need to pass Turborepo credentials as build secrets (not build arguments) to your container build for the following reasons: +When using `depot/build-push-action` to build Docker images that contain Turborepo workspaces, your build needs access to Turborepo's remote cache credentials to benefit from caching. -- Build secrets avoid cache invalidation because the values of `TURBO_API` and `TURBO_TOKEN` change. -- Build secrets keep tokens secure and prevent them from persisting in the final image. +These credentials are not automatically available inside your Docker build environment. Unlike builds running directly on Depot-managed GitHub Actions runners (which have automatic access to Depot Cache environment variables), containerized builds execute in isolated VMs that require explicit configuration. -Follow these steps to add your Turborepo credentials as build secrets. +Follow these steps to securely pass your Turborepo credentials into your Docker build: 1. Create GitHub Secrets for your Turborepo cache variables: - `TURBO_API` @@ -67,8 +62,7 @@ Follow these steps to add your Turborepo credentials as build secrets. 3. Update your Dockerfile to mount the secrets as environment variables: ```dockerfile -syntax=docker/dockerfile:1 -# ^ Required to use secret mounts with environment variables +# syntax=docker/dockerfile:1 # ... other Dockerfile instructions @@ -78,3 +72,5 @@ RUN --mount=type=secret,id=TURBO_API,env=TURBO_API \ --mount=type=secret,id=TURBO_TEAM,env=TURBO_TEAM \ turbo build ``` + +Adding `# syntax=docker/dockerfile:1` as the first line of your Dockerfile enables mounting secrets as environment variables. diff --git a/content/container-builds/build-parallelism.mdx b/content/container-builds/build-parallelism.mdx new file mode 100644 index 0000000..cce5eaa --- /dev/null +++ b/content/container-builds/build-parallelism.mdx @@ -0,0 +1,206 @@ +--- +title: Build parallelism in Depot +ogTitle: How build parallelism works in Depot +description: Learn how BuildKit's parallel execution works across build stages, multi-platform builds, and concurrent builds to maximize build speed and efficiency. +--- + +Depot uses BuildKit under the hood, which features a fully concurrent build graph solver that can run build steps in parallel when possible and optimize out commands that don't have an impact on the final result. This means that independent build stages, layers, and even separate builds can execute simultaneously. Understanding how parallelization works across different scenarios helps you structure your builds for maximum efficiency and speed. + +## Choosing the right build configuration + +Before diving into how parallelism works, it's important to understand the optimal build configuration for your workload. Depot offers several configuration options to balance performance, cache utilization, and resource allocation based on your specific needs. + +**Configuration decision matrix:** + +| Workload type | Recommended configuration | Reasoning | +| ---------------------------------------- | ----------------------------------------------------------- | ----------------------------------------- | +| Frequent small builds | Larger builder instance, no auto-scaling | Better cache utilization | +| Resource-intensive builds | Auto-scaling with Builds per instance = 2-3 | Each build gets full resources | +| Mixed workloads | Use separate projects per target | Balance between isolation and cache | +| Monorepo with shared dependencies (Bake) | Enable auto-scaling and/or use separate projects per target | Balance deduplication with resource needs | + +## Parallelism scenarios + +### One build per project + +When you run a single build in a Depot project, parallelism occurs at multiple levels: + +#### Stage-level parallelism + +If BuildKit sees that a stage depends on other stages which do not depend on each other, then it will run those stages in parallel. Consider this Dockerfile: + +```dockerfile +FROM node:20 AS frontend +WORKDIR /app +COPY frontend/ . +RUN npm install && npm build + +FROM golang:1.21 AS backend +WORKDIR /app +COPY backend/ . +RUN go build -o server + +FROM alpine AS final +COPY --from=frontend /app/dist /static +COPY --from=backend /app/server /usr/bin/ +``` + +Build execution flow: + +![Stage level parallelism](/images/docs/stage-level-parallelism.excalidraw.svg) + +In this example, the `frontend` and `backend` stages run in parallel since they don't depend on each other. The `final` stage waits for both to complete. + +#### Multi-platform parallelism + +When building for multiple platforms (e.g., `linux/amd64` and `linux/arm64`), Depot runs native builders for each architecture in parallel. Each platform executes on its own dedicated build server with native CPU architecture, which enables true parallel builds at native speed. + +```bash +# Builds for both platforms simultaneously on separate native servers +depot build --platform linux/amd64,linux/arm64 . +``` + +![Multi-platform build architecture](/images/docs/multi-platform-build-architecture.excalidraw.svg) + +### Multiple builds per project + +Each Depot project has dedicated BuildKit runners, with one runner per architecture by default. For example, if you're building for both `linux/amd64` and `linux/arm64`, you get two runners. All builds on the same architecture share that architecture's runner, enabling BuildKit to handle concurrent builds efficiently, whether they're for the same image or different images. + +![Multiple concurrent builds on same builder](/images/docs/multiple-concurrent-builds.excalidraw.svg) + +This shared runner architecture enables several optimizations: + +**Same image, multiple builds:** +When multiple builds of the same image run concurrently (e.g., different developers pushing to the same branch), BuildKit can: + +- Share cached layers across all builds +- Deduplicate identical work happening simultaneously +- Reduce overall build time through shared computation + +**Different images, shared dependencies:** +When building different images that share common dependencies: + +- Base images are pulled once and shared +- Common layers (like `npm install` or `apt-get update`) are computed once +- BuildKit automatically identifies and shares identical work + +#### BuildKit deduplication + +BuildKit's deduplication is a key optimization that automatically identifies and eliminates redundant work. BuildKit uses checksums to identify identical layers and operations through content-addressable storage. The build graph solver identifies duplicate work before execution, and when multiple stages need the same layer, it's built once and shared. Examples of deduplication include the following: + +- Multiple stages using the same base image only pull it once +- Repeated `RUN` commands with identical inputs are executed once +- Common file copies across stages are cached and reused + +![BuildKit deduplication within a build](/images/docs/buildkit-deduplication.excalidraw.svg) + +```dockerfile +FROM node:20 AS deps +COPY package*.json ./ +RUN npm ci # This layer is built once + +FROM node:20 AS deps +COPY package*.json ./ +RUN npm ci # Reuses the layer from Service A if cache is warm +``` + +In the preceding example, if both stages have identical `package.json` files, BuildKit recognizes that the `npm ci` command will produce the same result. Instead of running it twice, it executes once and reuses the cached layer for the second stage, saving build time and resources. + +This cache-based deduplication happens automatically across concurrent builds on the same runner, for builds triggered in any of the following ways: + +- Multiple `depot build` commands +- `depot bake` with multiple targets +- Parallel CI/CD jobs +- Multiple developers building simultaneously the same Dockerfile + +**Waiting for shared layers** + +When the same instruction is being built multiple times on the same runner, you may notice delays even with high cache hit rates. The delay is due to BuildKit's step deduplication process: one build computes the step while others wait for it to complete. This process prevents redundant work but can cause apparent delays. Subsequent builds show as "waiting" even though they'll benefit from the computed result. + +![Cross-build deduplication timeline](/images/docs/cross-build-deduplication-timeline.excalidraw.svg) + +When Build A starts building at 10:00 AM, it pulls the base image and runs `npm ci`, creating new layers. When Build B starts building just a minute later at 10:01 AM, BuildKit recognizes that it needs the same base image and has the same `npm ci` command. Instead of duplicating this work, Build B waits for Build A to complete those steps, then reuses the layers that Build A created. + +The deduplication process generally improves overall efficiency, but can be confusing when monitoring individual build times. To avoid overwhelming a single build server, you can enable [build auto-scaling](./how-to-guides/autoscaling) to some particular maximum parallelism value. + +#### Docker Bake for orchestrated builds + +Docker Bake provides a declarative way to build multiple images with a single command, taking full advantage of BuildKit's parallelism. By default, all Bake targets run on the same builder, which maximizes cache sharing and deduplication but means all targets share the same resources. + +Here's an example `docker-bake.hcl` configuration: + +```hcl +group "default" { + targets = ["app", "db", "cron"] +} + +target "base" { + dockerfile = "Dockerfile.base" + tags = ["myrepo/base:latest"] + project-id = "project-base" +} + +target "app" { + contexts = { + base = "target:base" + } + dockerfile = "Dockerfile.app" + platforms = ["linux/amd64", "linux/arm64"] + tags = ["myrepo/app:latest"] + project-id = "project-app" +} + +target "db" { + contexts = { + base = "target:base" + } + dockerfile = "Dockerfile.db" + platforms = ["linux/amd64", "linux/arm64"] + tags = ["myrepo/db:latest"] + project-id = "project-db" +} + +target "cron" { + contexts = { + base = "target:base" + } + dockerfile = "Dockerfile.cron" + platforms = ["linux/amd64", "linux/arm64"] + tags = ["myrepo/cron:latest"] + project-id = "project-cron" +} +``` + +When you run `depot bake`, all three services (`app`, `db`, `cron`) build concurrently for both architectures. With the `project-id` parameters specified, each target gets its own dedicated builder with separate resources. The base image is built once on its own project and the result is shared across the other targets via the contexts configuration. + +![Bake: Shared project vs separate projects](/images/docs/bake-shared-project-vs-separate-projects.excalidraw.svg) + +### Auto-scaling enabled + +With build auto-scaling enabled, Depot will automatically spin up additional BuildKit builders when the concurrent build limit is reached. By default, all builds for a project are routed to a single BuildKit host per architecture you're building. When the concurrent build limit is reached, Depot provisions additional builders. Each additional builder operates on a clone of the main builder's layer cache. + +![Auto-scaling behavior](/images/docs/auto-scaling-behavior.excalidraw.svg) + +Benefits: + +- Each build gets dedicated resources (CPU, memory, I/O) +- No resource contention between builds +- Consistent, predictable build times +- Better for resource-intensive builds + +Trade-offs: + +- Additional builders operate on cache clones that are not written back to the main cache, meaning work done on additional builders must be recomputed when subsequent builds run on the main builder +- Builds on different builders cannot share work, even if they have similar layers + +#### Configuration + +For detailed instructions on enabling and configuring auto-scaling, see the [Auto-scaling documentation](./how-to-guides/autoscaling). + +**Poor cache performance with auto-scaling** + +Cache misses are expected behavior with cache clones. Consider if the speed benefit outweighs cache efficiency. Try the following solutions for poor cache performance: + +- Increase **Builds per instance** in your **Autoscaling** settings +- Use a larger single instance instead of scaling out +- If building multiple different images, consider using a separate Depot project for each image to isolate their caches and runners diff --git a/content/container-builds/how-to-guides/private-registries.mdx b/content/container-builds/how-to-guides/private-registries.mdx index 34df213..4eeec42 100644 --- a/content/container-builds/how-to-guides/private-registries.mdx +++ b/content/container-builds/how-to-guides/private-registries.mdx @@ -8,7 +8,7 @@ description: Learn how to build images with Depot that need to access private re Our `depot` CLI uses your local Docker credentials provider. So, any registry you've logged into with `docker login` or similar will be available when running a Depot build. This means that you can build images that use private registries like the example below. -```Dockerfile +```dockerfile FROM my-private-registry/project/image:version ... ``` diff --git a/content/container-builds/quickstart.mdx b/content/container-builds/quickstart.mdx index 04f26e5..8651b46 100644 --- a/content/container-builds/quickstart.mdx +++ b/content/container-builds/quickstart.mdx @@ -6,31 +6,33 @@ description: Get started with Depot for up to 40x faster container image builds Get faster container image builds by replacing `docker build` with `depot build`. -## Install the Depot CLI +## Prerequisites -You'll need the [Depot CLI](/docs/cli/reference) on your machine to run local builds. +You'll need a [Depot account](https://depot.dev/sign-up). -### macOS +## Install the Depot CLI -Install the Depot CLI with Homebrew: +Install the [Depot CLI](/docs/cli/reference) on your machine to run local builds. -```shell -brew install depot/tap/depot -``` +- **macOS** -### Linux + Install the Depot CLI with Homebrew: -Install the Depot CLI with the installation script. + ```shell + brew install depot/tap/depot + ``` -Install the latest version: +- **Linux** -```shell -curl -L https://depot.dev/install-cli.sh | sh -``` + Install the Depot CLI with the installation script: + + ```shell + curl -L https://depot.dev/install-cli.sh | sh + ``` -### All platforms +- **All platforms** -Download the binary file for your platform from the [Depot CLI releases page](https://github.com/depot/cli/releases) in GitHub. + Download the binary file for your platform from the [Depot CLI releases page](https://github.com/depot/cli/releases) in GitHub. ## Run a local build diff --git a/content/github-actions/how-to-guides/access-private-resources.mdx b/content/github-actions/how-to-guides/access-private-resources.mdx new file mode 100644 index 0000000..8540793 --- /dev/null +++ b/content/github-actions/how-to-guides/access-private-resources.mdx @@ -0,0 +1,114 @@ +--- +title: Access private resources from Depot runners +ogTitle: How to access private resources from Depot GitHub Actions runners +description: Learn how to securely connect Depot runners to your private resources. +--- + +Depot GitHub Actions runners can access your private resources, like internal APIs, databases, or other services. + +By default, each Depot runner launches with a unique public IP address from the AWS (Amazon Web Services) address pool. This approach has the following implications: + +- Every build job gets a different public IP address. +- Third-party services that rate-limit by IP address generally don't rate-limit your jobs because they have different IP addresses. +- You can't allowlist Depot runners by IP address to access private resources. + +## Do you really need static IP addresses for your Depot runners? + +If you need to securely access your private resources from your Depot runner, then we recommend choosing from several alternatives to static IP addresses. In general, options like placing your runners on your VPN or VPC peering (if you use AWS) perform better, are easier to maintain, and cost less. + +## How to access private resources securely + +We recommend the following approaches to securely connect Depot runners to your private resources. You can adapt these patterns to your infrastructure and requirements. + +### Tailscale integration + +The Tailscale integration allows Depot CI runners and container build runners to join your private Tailscale network, giving them secure access to internal resources without any infrastructure changes. + +When to use this approach: + +- You're already using Tailscale or are willing to adopt it. +- You need to access private resources across different cloud providers. +- You don't want to make any significant infrastructure changes. + +How it works: + +- Runners automatically join your Tailscale network at the start of each build. +- You configure access rules in your Tailscale ACL to control which resources runners can access. + +Setup: + +- For setup instructions, see the [Tailscale integration documentation](https://depot.dev/docs/integrations/tailscale). + +### Cloudflare Warp + +If you're using Cloudflare for authentication and access control, you can install Cloudflare Warp within your CI runners to give them a verifiable identity in your Zero Trust configuration. + +When to use this approach: + +- You're already using Cloudflare Zero Trust for your private resources. +- You want identity-based access control rather than IP-based allowlisting. + +How it works: + +- Set up Cloudflare Warp in your GitHub Actions workflow using the [setup-cloudflare-warp action](https://github.com/marketplace/actions/setup-cloudflare-warp). +- Runners receive a Cloudflare identity that you can reference in your Zero Trust policies. +- Control access to internal services through your Cloudflare Zero Trust configuration. + +Example usage: + +```yaml +steps: + - uses: actions/checkout@v4 + - uses: cloudflare/warp-action@v1 + with: + organization: your-org + - run: curl https://internal-service.example.com +``` + +### VPC peering with AWS + +For AWS-based infrastructure, Depot can establish direct VPC peering between your AWS account and the VPC where your runners operate. Depot configures a peering connection between VPCs, allowing runners to access resources in your private subnets and ensuring that traffic stays within the AWS network. + +When to use this approach: + +- Your private resources are in AWS. +- You need low-latency access to AWS resources. +- You prefer AWS-native networking solutions. + +Requirements: + +- Depot [Business plan](https://depot.dev/pricing) +- Existing AWS infrastructure + +## Static IP addresses + +If you have an absolute requirement for static IP addresses, you can consider upgrading to our Business plan for a custom deployment. + +Depot provisions dedicated infrastructure with dedicated VPCs for your runners. All runners either peer with your AWS account or are configured to use a NAT gateway for static outbound IP addresses. These IPs can then be allowlisted in your firewall or security policies. + +When to use this approach: + +- None of the VPN-based options (Tailscale, Cloudflare Warp) work for your security policies. +- VPC peering isn't applicable (non-AWS infrastructure). + +Tradeoffs compared to our default IP addressing model: + +- Traffic is limited by the NAT gateway's bandwidth. +- Since all builds share the same IP addresses, third-party services like Docker Hub are more likely to rate-limit your requests. +- Requires dedicated infrastructure and NAT gateway resources. + +Requirements: + +- Depot [Business plan](https://depot.dev/pricing) + +Deployment options: + +- Custom deployment with dedicated infrastructure in our cloud. +- Depot Managed deployment in your AWS organization. + +## Get help + +Reach out if you're not sure which option is right for your use case. + +- Join our [Discord community](https://discord.gg/depot) to ask questions and see what other developers are doing with Depot. +- [Contact us](mailto:help@depot.dev) for help or to learn more about plans and options. diff --git a/content/github-actions/quickstart.mdx b/content/github-actions/quickstart.mdx index 9c01516..54ae84e 100644 --- a/content/github-actions/quickstart.mdx +++ b/content/github-actions/quickstart.mdx @@ -6,9 +6,13 @@ description: Get faster GitHub Actions with Depot's fully-managed GitHub Actions Connect your Depot organization to GitHub and configure your GitHub Actions to use Depot managed runners. +## Prerequisites + +You'll need a [Depot account](https://depot.dev/sign-up). + ## Connect to GitHub -To configure Depot GitHub Action Runners, connect to your GitHub organization and install the Depot GitHub App: +To configure Depot GitHub Action Runners, you must have the organization owner role. Connect to your GitHub organization and install the Depot GitHub App: 1. Log in to your [Depot dashboard](/orgs). 2. Click **GitHub Actions**.