Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: add cli download to download public node snapshots #13598

Open
wants to merge 12 commits into
base: main
Choose a base branch
from
Open

feat: add cli download to download public node snapshots #13598

wants to merge 12 commits into from
+393 −127

Conversation

lean-apple
Copy link
Contributor

@lean-apple lean-apple commented Dec 30, 2024 •
edited
Loading

Closes #13469.

Example of use :

cargo run --bin reth -- download --url https://downloads.merkle.io/reth-2024-10-23.tar.lz4

Starting snapshot download for chain: Chain::Named(Mainnet)
Target directory: "/Users/xxxx/Library/Application/reth/mainnet"
Source URL: https://downloads.merkle.io/reth-2024-10-23.tar.lz4
Downloading... 0.1%

@codspeed-hq CodSpeed HQ
Copy link

codspeed-hq bot commented Dec 30, 2024 •
edited
Loading

CodSpeed Performance Report

Merging #13598 will not alter performance

Comparing lean-apple:cli-download-public-node-snapshots (74950e9) with main (4a8c88f)

Summary

✅ 77 untouched benchmarks

gakonst
gakonst reviewed Jan 6, 2025
View reviewed changes
Copy link
Member

@gakonst gakonst left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's also maybe add a list of pre-set snapshot URLs so that the user doesn't need to find the URL themselves? And we can default to one of them, while the --help menu shows alternatives/fallbacks?

@lean-apple lean-apple marked this pull request as ready for review January 6, 2025 16:37
@lean-apple
Copy link
Contributor Author

Let's also maybe add a list of pre-set snapshot URLs so that the user doesn't need to find the URL themselves? And we can default to one of them, while the --help menu shows alternatives/fallbacks?

What would you advise me to be able to fetch the same block height they are using in their url ? @matias-gonz @joshieDo
For example, in this url https://snapshots.publicnode.com/ethereum-holesky-reth-3087369.tar.lz4, we see it's3087369,
but if I use let latest_block = provider.get_block_number().await?; I got 3090694.

My aim is to build a default "dynamic" url with the corresponding block height with this pattern :
https://snapshots.publicnode.com/ethereum-{potential network}-reth-{block height}.tar.lz4.

joshieDo
joshieDo reviewed Jan 7, 2025
View reviewed changes
crates/cli/commands/src/download.rs
Comment on lines +48 to +50
println!("Starting snapshot download for chain: {:?}", self.chain.chain());
println!("Target directory: {:?}", data_dir.data_dir());
println!("Source URL: {}", self.url);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should use info! instead

joshieDo
joshieDo reviewed Jan 7, 2025
View reviewed changes
crates/cli/commands/src/download.rs
pub async fn execute<N>(self) -> Result<()> {
let data_dir = self.datadir.resolve_datadir(self.chain.chain());
let snapshot_path = data_dir.data_dir().join(SNAPSHOT_FILE);
fs::create_dir_all(&data_dir).await?;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

std::fs instead of tokio::fs

joshieDo
joshieDo reviewed Jan 7, 2025
View reviewed changes
crates/cli/commands/src/download.rs
let mut response = client.get(url).send().await?.error_for_status()?;

let total_size = response.content_length().unwrap_or(0);
let mut file = fs::File::create(&target_path).await?;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason for using tokio::fs::File here ?

joshieDo
joshieDo requested changes Jan 7, 2025
View reviewed changes
Copy link
Collaborator

@joshieDo joshieDo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Snapshots can be quite big (>2TB with an archival node).

The current PR would mean that we'd need double that space, since we're downloading everything first and then decompressing. Ideally we'd pipe what we download straight into the decompressing stage.

Basically replicating the following behaviour:
wget -O - https://downloads.merkle.io/reth-2025-01-06.tar.lz4 | tar -I lz4 -xvf -

@joshieDo
Copy link
Collaborator

joshieDo commented Jan 7, 2025 •
edited
Loading

Regarding merkle url, it's possible to find the latest archive with the following link: https://downloads.merkle.io/latest.txt

@lean-apple
Copy link
Contributor Author

lean-apple commented Jan 9, 2025 •
edited
Loading

Snapshots can be quite big (>2TB with an archival node).

The current PR would mean that we'd need double that space, since we're downloading everything first and then decompressing. Ideally we'd pipe what we download straight into the decompressing stage.

Basically replicating the following behaviour: wget -O - https://downloads.merkle.io/reth-2025-01-06.tar.lz4 | tar -I lz4 -xvf -

Thanks, I've added this option --decompress that indeed makes the downloading not opitmized, will update the code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gakonst gakonst gakonst left review comments

@joshieDo joshieDo joshieDo requested changes

@onbjerg onbjerg Awaiting requested review from onbjerg onbjerg is a code owner

@mattsse mattsse Awaiting requested review from mattsse mattsse is a code owner

Requested changes must be addressed to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add new cli command to download public node snapshots
3 participants
@lean-apple @joshieDo @gakonst