Interleave downloads with installations when downloading a toolchain #4471

FranciscoTGouveia · 2025-09-05T15:33:38Z

Continuation of #4436.

This change aims to speed up the overall performance of the rustup [toolchain] install command by allowing the installation of a component to start immediately after its download completes.
This approach is particularly beneficial for slower connections, as it enables installations to progress in parallel with ongoing downloads -- so by the time all downloads are finished, all installations should be completed as well.

NB: This PR is not yet ready for review. I believe this change should be backed by benchmarks and user feedback before moving forward. I’ll share those here in the next couple of days.

src/dist/manifestation.rs

tests/suite/cli_rustup.rs

src/dist/notifications.rs

src/cli/download_tracker.rs

src/dist/manifestation.rs

FranciscoTGouveia · 2025-09-17T22:10:34Z

As noted in the opening message of the PR, I now present some benchmarks that will enable us to move forward with this change.
As stated before, this will bump the install and update commands' performance, as when the downloads finish, the installations have already executed.

The results are presented in the plot below, for both low-speed and high-speed internet connections.

These results were obtained by running the rustup toolchain install nightly command 10x times for each combination of variables (RUSTUP_CONCURRENT_DOWNLOADS, interleaved or not, and internet connection).
Please note that the y axis do not start on zero, and cannot be directly compared between themselves.

Below, I also leave a small animation of the previous and current behavior of the rustup toolchain install command to illustrate the UX changes that this PR also implies.
On the left you can see the previous version and on the right, the version with the interleaved installations.
Please note that this is just an animation and does not have the purpose of showing any kind of speedup.

Thanks for taking the time to review this PR!

src/config.rs

src/dist/component/package.rs

src/dist/download.rs

src/dist/temp.rs

rami3l

Modulo the previously requested changes and the two extra nits, I think this PR is quite ready. Thanks a lot, @FranciscoTGouveia!

src/dist/manifestation.rs

src/cli/common.rs

src/dist/manifestation.rs

FranciscoTGouveia · 2025-10-18T11:05:33Z

Hello everyone!

I think all review comments should be addressed by now with this and this comments being left for a future PR -- the justification can be seen in the comments.

I suspect some interaction with memory limits showing up in the unpack phase there; I suggest raising the working memory much higher, if you're on a machine that has capacity, to see if thats the case.

But also it suggests the default should probably be quite conservative, as we still need to support unpacking on Rasberry PI class machines.

First of all, thank you for your comment @rbtcollins. I have indeed reran the benchmarks with a higher memory and it didn't show any difference. For reference, it was a machine equipped with an Intel Xeon E5-2630 v2 processor and 64 GB of RAM.

Moreover, I would like to thank @djc for all the simplification work that he did, which made this PR much easier to implement. 😄

rami3l

LGTM modulo some questions/nits :)

src/config.rs

src/dist/component/transaction.rs

src/dist/manifestation.rs

djc

So while I did say I was fine with doing spawning/channel approach, after looking at the "remove some lifetime parameters" commit I am no longer sure if that's the best approach.

Effectively it means that whoever goes and puts in place the right architecture will basically have to undo all of those changes which IMO should not be necessary.

src/dist/manifestation.rs

djc · 2025-10-18T14:30:22Z

(It might be useful to start split some of the commits out of this PR, since it's obviously complex/contentious -- would be useful to start landing some parts to avoid more rebasing. At least the first commit is an obvious candidate.)

djc · 2025-10-20T07:41:02Z

I think the correcting faulty output commits are also independent of the other tricky parts?

FranciscoTGouveia · 2025-10-20T07:46:57Z

I think the correcting faulty output commits are also independent of the other tricky parts?

Yes, they are. Would it be okay if I opened a PR for that now to help streamline this one?

djc · 2025-10-20T08:08:06Z

Yes, they are. Would it be okay if I opened a PR for that now to help streamline this one?

Definitely!

Even though downloads are done concurrently, the installations are done sequentially. This means that, as downloads complete, they are in a queue (an mpsc channel) waiting to be consumed by the future responsible for the (sequential) installations. There was a need to relax some test cases to allow for uninstall to happen before the downloads.

…hread installations Co-authored-by: rami3l <rami3l@outlook.com>

…nue during installations Co-authored-by: rami3l <rami3l@outlook.com>

…red` As installations are still done sequentially, we do not need to have a mpsc channel and can instead resort to a `FuturesUnordered`.

FranciscoTGouveia force-pushed the install-after-download branch from c4b68b2 to c2954ba Compare September 5, 2025 16:02

rami3l self-assigned this Sep 6, 2025

rami3l mentioned this pull request Sep 7, 2025

fix(cli/rustup_mode): use ASCII-compatible spinner #4473

Merged

FranciscoTGouveia force-pushed the install-after-download branch from c2954ba to 1d7e5f5 Compare September 10, 2025 13:27

rami3l reviewed Sep 10, 2025

View reviewed changes

FranciscoTGouveia force-pushed the install-after-download branch 4 times, most recently from 0ea9162 to a2a2c20 Compare September 15, 2025 22:14

rami3l reviewed Sep 16, 2025

View reviewed changes

src/dist/manifestation.rs Outdated Show resolved Hide resolved

FranciscoTGouveia force-pushed the install-after-download branch 2 times, most recently from 947b8da to 9e5d0e0 Compare September 16, 2025 10:40