Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add parallel pull mode #256

Merged
merged 27 commits into from
Jun 27, 2019
Merged

Add parallel pull mode #256

Show file tree
Hide file tree
Changes from 26 commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
71ff23d
WIP continuous sync.
artem-zinnatullin Apr 8, 2018
bad4749
Sleep between syncs.
artem-zinnatullin Apr 8, 2018
69b1b52
500 millis between syncs.
artem-zinnatullin Apr 8, 2018
806a9f1
Merge branch 'development' into az/continuous-sync
artem-zinnatullin Jul 30, 2018
56e3f7a
Fixes
artem-zinnatullin Nov 28, 2018
474c263
More progress on it
artem-zinnatullin Dec 4, 2018
b5d5610
merge master
artem-zinnatullin Apr 28, 2019
2ed4aeb
Switch to crossbeam channels
artem-zinnatullin Apr 28, 2019
0aa3a78
Add PullMode to config
artem-zinnatullin Apr 28, 2019
4622234
Handle warnings
artem-zinnatullin Apr 28, 2019
06902f8
Fix clippy
artem-zinnatullin Apr 28, 2019
a5758a6
Add separate type for remote command result
artem-zinnatullin May 1, 2019
f85ca20
my god it finally compiles yooo
artem-zinnatullin May 1, 2019
7c655da
Rename sync_ stuff to push and pull
artem-zinnatullin May 1, 2019
6572462
Run all tests in two pull modes
artem-zinnatullin May 1, 2019
fe31d87
Merge branch '3.x' of github.com:buildfoundation/mainframer into az/c…
artem-zinnatullin May 1, 2019
be3b6a9
Add docs
artem-zinnatullin May 1, 2019
419b36c
Add tests with lots of big files, improve output
artem-zinnatullin May 1, 2019
3d39438
Improve result types
artem-zinnatullin May 4, 2019
69bdf25
Remove RemoteCommandResult
artem-zinnatullin May 4, 2019
8149b55
Simplify exit
artem-zinnatullin May 5, 2019
2e972d9
Remove sleep, fix wrong duration passed to calculate_perceived_pull_d…
artem-zinnatullin May 5, 2019
3340f3e
Replace while … with loop
artem-zinnatullin May 5, 2019
8722dd1
Merge branch '3.x' of github.com:buildfoundation/mainframer into az/c…
artem-zinnatullin May 5, 2019
2129174
ls files in test/test_pulls_big_files.sh
artem-zinnatullin May 5, 2019
e8ad1e2
merge 3.x
artem-zinnatullin Jun 27, 2019
b905d73
Update test after merge w/ upstream
artem-zinnatullin Jun 27, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
288 changes: 281 additions & 7 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,3 +11,5 @@ authors = ["Artem Zinnatullin <artem.zinnatullin@gmail.com>", "Artur Dryomov <ar
[dependencies]
yaml-rust = "0.4.2"
linked-hash-map = "0.5.1"
crossbeam-channel = "0.3"
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't used it in a while, but any reason not to use std::sync::mpsc?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Initially I swapped to crossbeam because I needed multi-receiver support, but then even that wasn't enough so I've also added bus

Decided to keep crossbeam for performance lol (nanoseconds matter hey!), we'll be stripping final binary #242 so it shouldn't matter which one we're using from the binary size point

bus = "2.2.1"
1 change: 1 addition & 0 deletions clippy.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
type-complexity-threshold = 450
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do you need it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

error: very complex type used. Consider factoring parts into `type` definitions
  --> src/sync.rs:93:47
   |
93 |     let (pull_finished_tx, pull_finished_rx): (Sender<Result<PullOk, PullErr>>, Receiver<Result<PullOk, PullErr>>) = unbounded();
   |                                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = note: `-D clippy::type-complexity` implied by `-D warnings`
   = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#type_complexity

error: very complex type used. Consider factoring parts into `type` definitions
   --> src/sync.rs:110:47
    |
110 |     let (pull_finished_tx, pull_finished_rx): (Sender<Result<PullOk, PullErr>>, Receiver<Result<PullOk, PullErr>>) = unbounded();
    |                                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#type_complexity

error: aborting due to 2 previous errors

😭

11 changes: 6 additions & 5 deletions docs/configuration/description.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,11 +30,12 @@ pull:
compression: {level}
```

Name | Required | Value | Default | Description
-------------------|----------|---------|---------|------------------
`remote.host` | Yes | `string`| — | Remote machine name from SSH config or hostname / IP address.
`push.compression` | No | `0..9` | `0` | Compression level used to copy files from local machine to remote one.
`pull.compression` | No | `0..9` | `0` | Compression level used to copy files from remote machine to local one.
Name | Required | Value | Default | Description
-------------------|----------|------------------------|---------|------------------
`remote.host` | Yes | `string` | — | Remote machine name from SSH config or hostname / IP address.
`push.compression` | No | `0..9` | `0` | Compression level used to copy files from local machine to remote one.
`pull.compression` | No | `0..9` | `0` | Compression level used to copy files from remote machine to local one.
`pull.mode` | No | `serial` \| `parallel` | `serial`| Pull mode. `serial` pulls once remote command has finished, `parallel` pulls in parallel to remote command execution.

Compression level is inherited from underlying `rsync`
which uses [`zlib` values](https://www.zlib.net/manual.html):
Expand Down
11 changes: 7 additions & 4 deletions src/config.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,21 +1,24 @@
#[derive(Debug, PartialEq)]
use sync::PullMode;

#[derive(Debug, PartialEq, Clone)]
pub struct Config {
pub remote: Remote,
pub push: Push,
pub pull: Pull,
}

#[derive(Debug, PartialEq)]
#[derive(Debug, PartialEq, Clone)]
pub struct Remote {
pub host: String,
}

#[derive(Debug, PartialEq)]
#[derive(Debug, PartialEq, Clone)]
pub struct Push {
pub compression: u8,
}

#[derive(Debug, PartialEq)]
#[derive(Debug, PartialEq, Clone)]
pub struct Pull {
pub compression: u8,
pub mode: PullMode,
}
1 change: 1 addition & 0 deletions src/ignore.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
use std::path::Path;
use std::path::PathBuf;

#[derive(Clone)]
pub struct Ignore {
pub common_ignore_file: Option<PathBuf>,
pub local_ignore_file: Option<PathBuf>,
Expand Down
Loading