Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don't store transaction data in JSON #314

Open
Endogen opened this issue Nov 3, 2024 · 0 comments
Open

Don't store transaction data in JSON #314

Endogen opened this issue Nov 3, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@Endogen
Copy link
Contributor

Endogen commented Nov 3, 2024

JSON isn't optimal for storing tx data since it doesn't preserve the order of keys and after decoding and getting a dict out of it, we need to order it first.

Better to use one of the following alternatives:

  1. MessagePack
    Faster than JSON for both encoding and decoding, while preserving the original structure, including key order.

Example code:

import msgpack

# Serialize data
packed_data = msgpack.packb(data_dict)

# Deserialize data
unpacked_data = msgpack.unpackb(packed_data)
  1. Protocol Buffers (Protobuf)
    Is a language-neutral and platform-neutral binary serialization format. Extremely efficient and supports schema definition, which ensures data consistency and improves compatibility. Highly optimized for performance and lower payload sizes.

Protobuf requires defining a .proto schema file and generating code, so it’s a bit more setup-heavy, but given that we already use it as part of CometBFT, we should probably switch to this.

@Endogen Endogen added the enhancement New feature or request label Nov 3, 2024
@xian-network xian-network deleted a comment from muizcal Nov 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant