Skip to content

CI: test Kubernetes port forwarder #4118

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Oct 18, 2025
Merged

CI: test Kubernetes port forwarder #4118

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions .github/workflows/test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -370,6 +370,19 @@ jobs:
template: templates/default.yaml
- name: "Run BATS integration tests"
run: make bats
- name: Cache image used by templates/k8s.yaml
uses: ./.github/actions/setup_cache_for_template
with:
template: templates/k8s.yaml
- name: "Run BATS k8s tests"
# Wish we could use BATS_TEST_RETRIES=3 as an environment variable here,
# but bats does not seem to support it.
uses: nick-fields/retry@ce71cc2ab81d554ebbe88c79ab5975992d79ba08 # v3.0.2
with:
timeout_minutes: 30
retry_on: error
max_attempts: 3
Comment on lines +378 to +384
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You have to set the variable inside the test that is flaky; the harness sets it to 0, so you can't set it from the outside:

❯ cat random.bats
@test "flaky test that fails 70% of the time" {
    BATS_TEST_RETRIES=10
    echo "Attempt $BATS_TEST_TRY_NUMBER/$BATS_TEST_RETRIES" >&3
    (( RANDOM % 10 < 3))
}

❯ ./hack/bats/lib/bats-core/bin/bats random.bats
random.bats
   flaky test that fails 70% of the time                                                                                                                             1/1
Attempt 1/10
 ✓ flaky test that fails 70% of the time
Attempt 2/10

1 test, 0 failures

❯ ./hack/bats/lib/bats-core/bin/bats random.bats
random.bats
   flaky test that fails 70% of the time                                                                                                                             1/1
Attempt 1/10
   flaky test that fails 70% of the time                                                                                                                             1/1
Attempt 2/10
   flaky test that fails 70% of the time                                                                                                                             1/1
Attempt 3/10
   flaky test that fails 70% of the time                                                                                                                             1/1
Attempt 4/10
   flaky test that fails 70% of the time                                                                                                                             1/1
Attempt 5/10
 ✓ flaky test that fails 70% of the time
Attempt 6/10

1 test, 0 failures

Without the logging you cannot tell if the test needed retries to pass (and BATS_TEST_RETRY_NUMBER is not defined in setup or teardown).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can set it at the file level, so it applies to every test in the same file:

BATS_TEST_RETRIES=10

@test "flaky test that fails 70% of the time" {
    echo "Attempt $BATS_TEST_TRY_NUMBER/$BATS_TEST_RETRIES" >&3
    (( RANDOM % 10 < 3))
}

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel GHA YAML is the right place to set BATS_TEST_RETRIES. Maybe we should submit a proposal to the upstream.

Can we just merge this PR?

command: ./hack/bats/lib/bats-core/bin/bats --timing ./hack/bats/extras/k8s.bats

colima:
name: "Colima tests (QEMU, Linux host)"
Expand Down
6 changes: 6 additions & 0 deletions hack/bats/extras/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# Extra tests

The extra tests located in this directory are not automatically executed via `make bats`.

Some tests are executed on the CI, some ones are not.
Refer to the configuration of the GitHub Actions to see what tests are executed.
64 changes: 64 additions & 0 deletions hack/bats/extras/k8s.bats
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
# SPDX-FileCopyrightText: Copyright The Lima Authors
# SPDX-License-Identifier: Apache-2.0

# This test verifies that a Kubernetes cluster can be started and that the single node is ready.

load "../helpers/load"

: "${TEMPLATE:=k8s}"

# Instance names are "${NAME}-0", "${NAME}-1", ...
NAME="k8s"

get_num_nodes() {
local nodes=0
for tag in "${BATS_TEST_TAGS[@]}"; do
if [[ $tag =~ ^nodes:([0-9]+)$ ]]; then
nodes="${BASH_REMATCH[1]}"
fi
done
Comment on lines +15 to +19
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not convinced that this is a good way to control setup for individual tests.

I think it would be clearer not to rely on local_setup, but to have each test call a setup function with an argument, like

@test 'Single-node configuration' {
   setup_nodes 1
   # ...
}
@test 'Multi-node configuration' {
   setup_nodes "${LIMA_BATS_K8S_NODES:-3}"
   # ...
}

And local_teardown could use limactl ls --quiet | grep "^${NAME}-" to get the list of instances to delete.

This also allows you to control the number of nodes from the outside without modifying the test source code to adjust the tag.

(Ab)using the tags to pass arguments to the setup/teardown functions is a bit confusing, and should be simpler. But we can always change this later, e.g. when there actually is a multi-node test.

if [[ $nodes -eq 0 ]]; then
echo >&2 "nodes:N tag is required"
exit 1
fi
echo "$nodes"
}

local_setup() {
local nodes=$(get_num_nodes)
for ((i=0; i<nodes; i++)); do
limactl delete --force "${NAME}-$i" || :
limactl start --tty=false --name "${NAME}-$i" "template:${TEMPLATE}" 3>&- 4>&-
# NOTE: No support for multi-node clusters yet.
done
for node in $(k get node -o name); do
k wait --timeout=5m --for=condition=ready "${node}"
done
}

local_teardown() {
local nodes=$(get_num_nodes)
for ((i=0; i<nodes; i++)); do
limactl delete --force "${NAME}-$i" || :
done
}

k() {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would call it kubectl for readability. Local functions always take precedence over running external programs.

# The host home directory is not mounted in the case of k8s.
limactl shell --workdir=/ "${NAME}-0" -- kubectl "$@"
}

# bats test_tags=nodes:1
@test 'Single-node' {
k create deployment nginx --image="${TEST_CONTAINER_IMAGES["nginx"]}"
k rollout status deployment nginx --timeout 60s
k create service nodeport nginx --node-port=31080 --tcp=80:80
run curl --fail --silent --show-error --retry 30 --retry-all-errors http://localhost:31080
assert_success
assert_output --partial "Welcome to nginx"
# TODO: support UDP
k delete service nginx
k delete deployment nginx
}

# TODO: add a test for multi-node
6 changes: 6 additions & 0 deletions hack/bats/helpers/load.bash
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,3 +65,9 @@ teardown() {
assert_output_lines_count() {
assert_equal "${#lines[@]}" "$1"
}

# Use GHCR and ECR to avoid hitting Docker Hub rate limit.
# NOTE: keep this list in sync with hack/test-templates.sh .
declare -A -g TEST_CONTAINER_IMAGES=(
["nginx"]="ghcr.io/stargz-containers/nginx:1.19-alpine-org"
)
2 changes: 1 addition & 1 deletion templates/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ Container orchestration:
- [`faasd`](./faasd.yaml): [Faasd](https://docs.openfaas.com/deployment/edge/)
- [`k0s`](./k0s.yaml): [k0s](https://k0sproject.io/) Zero Friction Kubernetes
- [`k3s`](./k3s.yaml): Kubernetes via k3s
- [`k8s`](./k8s.yaml): Kubernetes via kubeadm
- [`k8s`](./k8s.yaml): ⭐Kubernetes via kubeadm
- [`experimental/u7s`](./experimental/u7s.yaml): [Usernetes](https://github.com/rootless-containers/usernetes): Rootless Kubernetes

Optional feature enablers:
Expand Down
Loading