Skip to content

Document how to debug performance of Prow jobs in etcd #20796

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Document how to debug performance of Prow jobs in etcd #20796

wants to merge 1 commit into from

Conversation

@ronaldngounou
Copy link
Member

Overview

This PR documents how etcd contributors can navigate performance of prow jobs.
Contributes to issue #18707

@k8s-ci-robot
Copy link

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@k8s-ci-robot
Copy link

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: ronaldngounou
Once this PR has been reviewed and has the lgtm label, please assign ivanvc for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ronaldngounou ronaldngounou force-pushed the debug-performance-prow branch from eeca3a0 to 2e38276 Compare October 15, 2025 07:59
@ronaldngounou ronaldngounou mentioned this pull request Oct 16, 2025
Open
@ronaldngounou ronaldngounou force-pushed the debug-performance-prow branch from 2e38276 to 6d3c4f2 Compare October 19, 2025 06:49
@ronaldngounou ronaldngounou marked this pull request as ready for review October 19, 2025 06:52
@ronaldngounou ronaldngounou force-pushed the debug-performance-prow branch 5 times, most recently from 822a203 to eb48c66 Compare October 19, 2025 07:22
@codecov
Copy link

codecov bot commented Oct 19, 2025 •
edited
Loading

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 69.24%. Comparing base (855c9ba) to head (5598b5d).

Additional details and impacted files

see 25 files with indirect coverage changes

@@            Coverage Diff             @@
##             main   #20796      +/-   ##
==========================================
+ Coverage   69.19%   69.24%   +0.04%     
==========================================
  Files         422      422              
  Lines       34822    34822              
==========================================
+ Hits        24095    24111      +16     
+ Misses       9331     9324       -7     
+ Partials     1396     1387       -9

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 855c9ba...5598b5d. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ronaldngounou
Copy link
Member Author

ronaldngounou commented Oct 22, 2025 •
edited
Loading

Hi @ivanvc , could you please review?
Is there anything I could do to resolve the failing tests ?

@serathius
Copy link
Member

Is there anything I could do to resolve thr failing tests ?

Please rebase

@ivanvc
Copy link
Member

ivanvc commented Oct 23, 2025

Hi @ronaldngounou, there are a couple of broken links in the document you pushed. Refer to the Prow job failure: https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/etcd-io_etcd/20796/pull-etcd-verify/1979810317445632000#

Found broken reference (Prow -> ) at /home/prow/go/src/github.com/etcd-io/etcd/Documentation/contributor-guide/prow_jobs.md:5
Found broken path (dashboards -> monitoring-gke.prow.k8s.io/d/96Q8oOOZk/builds?orgId=1&refresh=30s&var-org=etcd-io&var-repo=etcd&var-build=All&from=now-7d&to=now) at /home/prow/go/src/github.com/etcd-io/etcd/Documentation/contributor-guide/prow_jobs.md:41

@ronaldngounou
Document how Prow jobs are used in etcd
5598b5d 
- Introduction to Prow
- How Prow is used for etcd testing
- Navigating perfomance dashboards (Grafana)
- Prow job categories
- Interpreting metrics

Signed-off-by: ronaldngounou <ronald.ngounou@yahoo.com>
@ronaldngounou ronaldngounou force-pushed the debug-performance-prow branch from eb48c66 to 5598b5d Compare October 25, 2025 02:58
@ronaldngounou
Copy link
Member Author

I have rebased and fixed the broken links.
cc @ivanvc @serathius

ivanvc
ivanvc reviewed Oct 26, 2025
View reviewed changes
Copy link
Member

@ivanvc ivanvc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, @ronaldngounou. This is a great start. I left some comments. Please take a look at them.

Documentation/contributor-guide/prow_jobs.md
Comment on lines +12 to +14
The CI of etcd is managed by kubernetes/test-infra, which leverages prow inside it.

Whenever a pull request is submitted, or a command is called, the CI of etcd, managed by [kubernetes/test-infra](https://github.com/kubernetes/test-infra) leverages Prow to run the tests. You can find all the supported [commands](https://prow.k8s.io/command-help).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These two paragraphs are very repetitive. I suggest consolidating into a single one.

Documentation/contributor-guide/prow_jobs.md

## 1. Introduction to Prow

[Prow](https://docs.prow.k8s.io/docs/) is a Kubernetes based CI/CD system. Jobs can be triggered by various types of events and report their status to many different services. In addition to job execution, Prow provides GitHub automation in the form of policy enforcement, chat-ops via `/foo` style commands, and automatic PR merging.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would emphasize that it responds to pull request comments using /commands.

Documentation/contributor-guide/prow_jobs.md

[Prow](https://docs.prow.k8s.io/docs/) is a Kubernetes based CI/CD system. Jobs can be triggered by various types of events and report their status to many different services. In addition to job execution, Prow provides GitHub automation in the form of policy enforcement, chat-ops via `/foo` style commands, and automatic PR merging.

When a user comments `/ok-to-test`or `/retest,` on a Pull Request, GitHub sends a webhook to Prow's Kubernetes cluster. The request travels through an ingress for TLS termination, gets routed to the hook service, and arrives at the hook application running in pods. Visit this [site](https://docs.prow.k8s.io/docs/life-of-a-prow-job/) to further understand the lifecycle of a Prow job.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This paragraph: ingress, TLS termination, etc, may be too specific. I don't know what's k8s prow's setup, but it's also irrelevant IMO to our use case.

Documentation/contributor-guide/prow_jobs.md

Whenever a pull request is submitted, or a command is called, the CI of etcd, managed by [kubernetes/test-infra](https://github.com/kubernetes/test-infra) leverages Prow to run the tests. You can find all the supported [commands](https://prow.k8s.io/command-help).

The [history](https://prow.k8s.io/job-history/gs/kubernetes-ci-logs/pr-logs/directory/pull-etcd-e2e-amd64?buildId=)of the ran can be found.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This paragraph looks unfinished, right?

Documentation/contributor-guide/prow_jobs.md

### How to manually run a given job on Prow

These tests can be triggered when you leave a comment, like `/ok-to-test` or `/retest`, in PR [example](https://github.com/etcd-io/etcd/pull/20733#issuecomment-3341443205).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Noting that /ok-to-test can only be triggered by an etcd-io member.

Documentation/contributor-guide/prow_jobs.md

## 3. Navigating Performance Dashboard (Grafana)

One example of [dashboards](https://monitoring-gke.prow.k8s.io/d/96Q8oOOZk/builds?orgId=1&refresh=30s&var-org=etcd-io&var-repo=etcd&var-job=ci-etcd-robustness-main-amd64&var-build=All&from=now-7d&to=now) can be found.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should clearly list the gke and eks Grafana dashboards, not just as examples.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@ivanvc ivanvc ivanvc left review comments

Reviewers whose approvals may not affect merge requirements

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

area/documentation size/L

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@ronaldngounou @k8s-ci-robot @serathius @ivanvc