Skip to content

Rust Migration

tenchd edited this page Aug 5, 2021 · 3 revisions

Investigating the potential gains of migrating our codebase to Rust at some point.
Benefits:

  • Rust is the future.
  • It's easier and more straightforward to write optimized code.
  • Many conveniences of modern languages which C++ lacks.

Downsides:

  • We have to migrate from C++ (maybe not so terrible since our codebase isn't huge)
  • May or may not be good distributed frameworks for Rust yet.
  • Many contributors are more used to C++.

As an informal experiment, I implemented the xor hashing data structure in Rust in an afternoon and optimized it slightly with a friend who's familiar with Rust. On my desktop, it's within a factor 3 of the ingestion speeds of our code on bigboy. I imagine that if we optimized it some more it'd be comparable.

Vector size 128 bit buckets 64 bit buckets xor hashing .5 bucket factor Rust xor hashing .5 reps (david's desktop)
10^3 101840.357 updates/s 174429.354 updates/s 7156454.406 updates/s 3989872.490 updates/s
10^4 55797.655 updates/s 96216.757 updates/s 6588353.109 updates/s 2975531.218 updates/s
10^5 25971.327 updates/s 45342.837 updates/s 6015725.105 updates/s 2346723.408 updates/s
10^6 17071.867 updates/s 25876.702 updates/s 5036844.517 updates/s 2121060.818 updates/s
10^7 12132.933 updates/s 18442.902 updates/s 4407694.070 updates/s 1780341.003 updates/s
10^8 9490.997 updates/s 14383.936 updates/s 4096379.619 updates/s 1539481.238 updates/s
10^9 7652.748 updates/s 11602.058 updates/s 3673283.474 updates/s 1445982.459 updates/s
10^10 1269.098 updates/s N/A 3296869.951 updates/s 1272333.978 updates/s
10^11 912.666 updates/s N/A 3146148.013 updates/s 1144202.545 updates/s
10^12 805.574 updates/s N/A 2888870.913 updates/s 1082557.372 updates/s

10^13 through 10^18: 995255.484735287 893804.316069317 867584.356333136 811561.536817464 799080.037052734 729061.895535921