skip CRC checksumming during diskless full sync with TLS enabled. #1479

talxsha · 2024-12-23T15:37:43Z

Implemented a mechanism to eliminate CRC64 checksumming during full sync when not writing to disk (using a connection that has data integrity checks such as TLS), as it adds overhead with minimal benefit.

Nodes can skip CRC calculations when these conditions are met:

Running diskless sync on primary.
Running diskless load on the replica.
Primary-replica connection is integrity checked.

Closes #1129

ranshid · 2024-12-23T19:04:51Z

@talxsha before I look into this, lets put some details in the top comment. linking the issue is not what we susually do.
Please state shortly what is the problem we are solving and what this solution includes.

src/replication.c

src/config.c

src/server.c

src/server.h

src/rdb.c

madolson · 2024-12-24T04:08:42Z

Added the functionality to disable CRC calculations during diskless full sync with TLS enabled.

Also add justification for why we should do this only when TLS is enabled. Given that the network has built in checksumming, I'm still not convinced about the tradeoff we are making given that the steady state replication is not checksummed.

ranshid · 2024-12-24T05:23:52Z

@madolson should I tag it as a major-decision ? I think it worth discussion.

src/replication.c

madolson · 2024-12-24T06:03:26Z

@madolson should I tag it as a major-decision ? I think it worth discussion.

For now it's not. It's just an internal one. I would probably just ping PingXie directly and core team if anyone else is interested.

Signed-off-by: Tal Shachar <talxsha@amazon.com>

…o bypass_crc, encapsulated condition checks for skipping CRC, and chenged connIsTLS condition to connIntegrityChecked in ConnectionType. Some changes in the test as well Signed-off-by: Tal Shachar <talxsha@amazon.com>

Signed-off-by: talxsha <160726520+talxsha@users.noreply.github.com>

src/connection.h

ranshid · 2025-01-02T09:51:59Z

src/server.h

@@ -1988,6 +1993,7 @@ struct valkeyServer {
    char *rdb_filename;                   /* Name of RDB file */
    int rdb_compression;                  /* Use compression in RDB? */
    int rdb_checksum;                     /* Use RDB checksum? */
+    int bypass_crc;                       /* Skip RDB checksum? Applicable only for TLS enabled diskless full sync */


I am not sure why do we need to keep this server flag?

src/replication.c

Signed-off-by: Tal Shachar <talxsha@amazon.com>

src/rdb.c

Signed-off-by: ranshid <88133677+ranshid@users.noreply.github.com>

ranshid

@madolson the code seems fine now. would you like to take a quick look as well?

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>

madolson

The code looks okay, mostly some nits to improve clarity.

src/tls.c

src/unix.c

src/server.c

src/socket.c

src/replication.c

tests/integration/bypass-crc.tcl

src/replication.c

src/rdb.c

src/replication.c

Signed-off-by: talxsha <160726520+talxsha@users.noreply.github.com>

…ecessary metric and added a log instead. Using sendCommandArgv when sendding replica capa Signed-off-by: Tal Shachar <talxsha@amazon.com>

codecov · 2025-01-17T06:09:47Z

Codecov Report

Attention: Patch coverage is 73.52941% with 9 lines in your changes missing coverage. Please review.

Project coverage is 71.00%. Comparing base (dc9ca1b) to head (5ea896f).
Report is 11 commits behind head on unstable.

Files with missing lines	Patch %	Lines
src/replication.c	66.66%	8 Missing ⚠️
src/rdb.c	85.71%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##           unstable    #1479      +/-   ##
============================================
+ Coverage     70.98%   71.00%   +0.02%     
============================================
  Files           120      120              
  Lines         65061    65090      +29     
============================================
+ Hits          46185    46219      +34     
+ Misses        18876    18871       -5

Files with missing lines	Coverage Δ
src/connection.h	`87.77% <100.00%> (+0.27%)`	⬆️
src/rdma.c	`100.00% <ø> (ø)`
src/rio.c	`84.83% <100.00%> (ø)`
src/rio.h	`100.00% <ø> (ø)`
src/server.h	`100.00% <ø> (ø)`
src/socket.c	`91.62% <ø> (ø)`
src/tls.c	`100.00% <ø> (ø)`
src/unix.c	`73.49% <ø> (ø)`
src/rdb.c	`77.03% <85.71%> (+0.09%)`	⬆️
src/replication.c	`87.31% <66.66%> (-0.14%)`	⬇️

... and 12 files with indirect coverage changes

ranshid · 2025-01-17T06:37:21Z

@talxsha please fix the spellcheck/format issues and also the tests should probably be tagged with cluster:skip

talxsha force-pushed the unstable branch from b13cec9 to 468729f Compare December 23, 2024 15:44

ranshid reviewed Dec 23, 2024

View reviewed changes

src/replication.c Outdated Show resolved Hide resolved

src/config.c Outdated Show resolved Hide resolved

src/server.c Outdated Show resolved Hide resolved

src/server.h Outdated Show resolved Hide resolved

ranshid reviewed Dec 23, 2024

View reviewed changes

src/rdb.c Outdated Show resolved Hide resolved

src/rdb.c Outdated Show resolved Hide resolved

ranshid reviewed Dec 24, 2024

View reviewed changes

src/replication.c Outdated Show resolved Hide resolved

talxsha marked this pull request as ready for review December 29, 2024 16:24

talxsha added 2 commits December 29, 2024 16:25

CRC removal during diskless full sync with TLS enabled

67fa361

Signed-off-by: Tal Shachar <talxsha@amazon.com>

talxsha force-pushed the unstable branch from cb3acfd to 2e5314e Compare December 29, 2024 16:25

Merge branch 'unstable' into unstable

f33f78f

Signed-off-by: talxsha <160726520+talxsha@users.noreply.github.com>

ranshid reviewed Jan 2, 2025

View reviewed changes

talxsha and others added 2 commits January 2, 2025 14:47

Merge branch 'valkey-io:unstable' into unstable

c605a83

Removed bypass_crc flag from server, renaming and comments changes

dd180dd

Signed-off-by: Tal Shachar <talxsha@amazon.com>

ranshid reviewed Jan 2, 2025

View reviewed changes

src/rdb.c Outdated Show resolved Hide resolved

src/rdb.c Outdated Show resolved Hide resolved

src/rdb.c Outdated Show resolved Hide resolved

src/rdb.c Outdated Show resolved Hide resolved

ranshid added 4 commits January 2, 2025 18:23

Update src/rdb.c

6126177

Signed-off-by: ranshid <88133677+ranshid@users.noreply.github.com>

Update src/rdb.c

5c34391

Signed-off-by: ranshid <88133677+ranshid@users.noreply.github.com>

Update src/rdb.c

e481f73

Signed-off-by: ranshid <88133677+ranshid@users.noreply.github.com>

Update src/rdb.c

14eeb6f

Signed-off-by: ranshid <88133677+ranshid@users.noreply.github.com>

ranshid approved these changes Jan 2, 2025

View reviewed changes

fix format issues

266cd65

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>

madolson reviewed Jan 2, 2025

View reviewed changes

talxsha and others added 3 commits January 5, 2025 11:22

Merge branch 'valkey-io:unstable' into unstable

be9bb4d

Merge branch 'unstable' into unstable

04e7dbe

Signed-off-by: talxsha <160726520+talxsha@users.noreply.github.com>

Changed bypass to skip and crc to checksum. Changed tests. Removed un…

c74edf0

…ecessary metric and added a log instead. Using sendCommandArgv when sendding replica capa Signed-off-by: Tal Shachar <talxsha@amazon.com>

talxsha force-pushed the unstable branch from 20609b0 to c74edf0 Compare January 12, 2025 09:57

Merge branch 'valkey-io:unstable' into unstable

5ea896f

ranshid mentioned this pull request Jan 14, 2025

[WIP] Skip the checksum for diskless replication #1181

Closed

ranshid changed the title ~~CRC removal during diskless full sync with TLS enabled.~~ skip CRC checksumming during diskless full sync with TLS enabled. Jan 14, 2025

madolson approved these changes Jan 15, 2025

View reviewed changes

madolson added the release-notes This issue should get a line item in the release notes label Jan 15, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

skip CRC checksumming during diskless full sync with TLS enabled. #1479

skip CRC checksumming during diskless full sync with TLS enabled. #1479

talxsha commented Dec 23, 2024 •

edited

Loading

ranshid commented Dec 23, 2024

madolson commented Dec 24, 2024

ranshid commented Dec 24, 2024

madolson commented Dec 24, 2024

ranshid Jan 2, 2025

ranshid left a comment

madolson left a comment

codecov bot commented Jan 17, 2025

ranshid commented Jan 17, 2025

skip CRC checksumming during diskless full sync with TLS enabled. #1479

Are you sure you want to change the base?

skip CRC checksumming during diskless full sync with TLS enabled. #1479

Conversation

talxsha commented Dec 23, 2024 • edited Loading

ranshid commented Dec 23, 2024

madolson commented Dec 24, 2024

ranshid commented Dec 24, 2024

madolson commented Dec 24, 2024

ranshid Jan 2, 2025

Choose a reason for hiding this comment

ranshid left a comment

Choose a reason for hiding this comment

madolson left a comment

Choose a reason for hiding this comment

codecov bot commented Jan 17, 2025

Codecov Report

ranshid commented Jan 17, 2025

talxsha commented Dec 23, 2024 •

edited

Loading