runtime: add runtime.Yield()

[dt]

This change is 

[petermattis]

I'm good with experimenting with this Go runtime enhancement. I definitely want to see experimental evidence of the benefit.

PS Should probably update the print in schedtrace to include sched.bgqsize.

[sumeerbhola]

Given that we are getting closer to merging this, I think we need a plan for how we will maintain this. Specifically the couple of lines of code scattered over various parts in proc.go e.g. we need a list of what cases we need to integrate with the changes, and how to go about finding those cases in the scheduler code.

[dt]

the couple of lines of code scattered over various parts in proc.go

Per the discussion thread above and in the google doc, I've cut these "lines scattered around" (assuming we're referring to the same ones) since my current approach is to just do searches of runqs when npidle is zero. This seems to perform about the same, and better handled added a check of netpoll which I realized I wanted anyway, so the diff is now much closer to a pure, more self contained addition: we need two additions to findRunnable, though they don't depend on anything other than the new yieldq, then two new fields: the yieldq in schedt and the yieldchecks counter/timestamp in G. Other than that it's just the pure addition of the Yield() function and its helpers now.

[dt]

I definitely want to see experimental evidence of the benefit.

Here's without/with making all Pacer.Pace() calls include a call to runtime.Yield() (even those where Pacer is nil/elastic AC is off) on a 5x8vcpu cluster running a TPCC 5k IMPORT while serving kv95 20k QPS in the foreground (note the units changed in the first one). CPU utilization is smoother and higher with Yields.

Without a foreground workload, IMPORT throughput is higher with Yields than without, somewhat surprisingly (by 5-10%) while it is slightly lower when there is a workload to yield to. CPU profiling indicate that while maintaining 98% CPU utilization, the Yield() calls account for about 0.8-0.9% of IMPORT's CPU usage and when under <90% utilization more like 0.4-0.6%.

[petermattis]

I like that this iteration is less intrusive than the previous one. Mostly comments about needing more comments.

[petermattis]

Is the expense of runqempty problematic? Seems like you're only doing this infrequently.

[dt]

Not that expensive (unless you have a silly number of cores) at least in relative terms in this branch where we have a whole syscall to netpoll. But I think it makes sense to skip it above when checking the local runq and if we're okay skipping it there, we should be okay skipping it here too (they can all be stolen from concurrently all the same).

[petermattis]

The results are very compelling.

[sumeerbhola]

@sumeerbhola reviewed 1 of 2 files at r3, all commit messages.
Reviewable status: 1 of 5 files reviewed, 25 unresolved discussions (waiting on @dt, @petermattis, and @tbg)

---

src/runtime/runtime2.go line 515 at r3 (raw file):

	runningnanos  int64 // wall time spent in the running state

	yieldchecks uint32 // a packed approx time and count of maybeYield checks.

This needs a longer code comment elaborating on what exactly this represents and what the packing scheme is.

---

src/runtime/proc.go line 420 at r3 (raw file):

		// To avoid thrashing between yields, set yieldchecks to 1: if we yield
		// right back and see this sentinel we'll park instead to break the cycle.
		gp.yieldchecks = 1

So sometimes yieldchecks is a packed field and sometimes 1? This needs code comments.

---

src/runtime/proc.go line 7251 at r3 (raw file):

}

// yield_put is the gopark unlock function for Yield. It enqueues the goroutine

I'm confused by the "unlock function" terminology, given there are callers of gopark that pass nil. And I want to make sure we document in a code comment why this is correct from the perspective of the following comment in gopark:

// unlockf must not access this G's stack, as it may be moved between
// the call to gopark and the call to unlockf.
//
// Note that because unlockf is called after putting the G into a waiting
// state, the G may have already been readied by the time unlockf is called
// unless there is external synchronization preventing the G from being
// readied. If unlockf returns false, it must guarantee that the G cannot be
// externally readied.

We don't access the stack, so fine. Is "readied" means transitioning back to runnable state? I suppose that can't happen either because we haven't even put it in the yieldq yet, so no one can discover it and make it runnable. Anything else I am missing?

[dt]

Reviewable status: 0 of 6 files reviewed, 25 unresolved discussions (waiting on @petermattis, @sumeerbhola, and @tbg)

---

src/runtime/proc.go line 7251 at r3 (raw file):

Previously, sumeerbhola wrote…

I'm confused by the "unlock function" terminology, given there are callers of gopark that pass nil. And I want to make sure we document in a code comment why this is correct from the perspective of the following comment in gopark:

// unlockf must not access this G's stack, as it may be moved between
// the call to gopark and the call to unlockf.
//
// Note that because unlockf is called after putting the G into a waiting
// state, the G may have already been readied by the time unlockf is called
// unless there is external synchronization preventing the G from being
// readied. If unlockf returns false, it must guarantee that the G cannot be
// externally readied.

We don't access the stack, so fine. Is "readied" means transitioning back to runnable state? I suppose that can't happen either because we haven't even put it in the yieldq yet, so no one can discover it and make it runnable. Anything else I am missing?

Yeah, that's pretty much it: nothing is going to ready this G until findRunnable pull it from the yieldq so it is our to pu there, so I think we can guarantee it isn't externally readied.

[dt]

Only one data point but I just rebased this from go 1.23 to go 1.25 since CRDB just switched today and the rebase was clean with no conflicts on the scaled back touch points of just findRunnable+two new fields. I think the previous version with the extra pending work atomic did have master/1.25 -> 1.23 conflicts when I went the other way, so maybe some evidence that being more contained is indeed helpful here.

[petermattis]

This approach looks good to me. I'll defer to @sumeerbhola for final approval. Really excited about the benefits you're seeing in test scenarios.

[petermattis]

Nit: anything parked?