Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Polkadart Milestone 1 #635

Merged
merged 1 commit into from
Dec 20, 2022
Merged

Polkadart Milestone 1 #635

merged 1 commit into from
Dec 20, 2022

Conversation

Lohann
Copy link
Contributor

@Lohann Lohann commented Nov 22, 2022

Milestone Delivery Checklist

Link to the application pull request: w3f/Grants-Program#1053

@Lohann Lohann force-pushed the polkadart-milestone-1 branch from 8413def to 56545dc Compare November 22, 2022 20:29
@alxs
Copy link
Contributor

alxs commented Nov 23, 2022

Thanks for the delivery @Lohann. We'll look into it as soon as possible.

@alxs alxs self-assigned this Nov 23, 2022
@alxs
Copy link
Contributor

alxs commented Nov 24, 2022

Hey again @Lohann, I'm your evaluator for this milestone. Thanks for the effort that went into this delivery, it looks great so far. A few comments from my side:

  • docker compose up to run the tests did not work for me:
    [+] Building 69.4s (17/17) FINISHED
     => [internal] load build definition from Dockerfile                                                                0.0s
     => => transferring dockerfile: 527B                                                                                0.0s
     => [internal] load .dockerignore                                                                                   0.0s
     => => transferring context: 2B                                                                                     0.0s
     => [internal] load metadata for docker.io/library/ubuntu:22.04                                                     1.1s
     => [internal] load build context                                                                                   0.0s
     => => transferring context: 2.71MB                                                                                 0.0s
     => [ 1/12] FROM docker.io/library/ubuntu:22.04@sha256:4b1d0c4a2d2aaf63b37111f34eb9fa89fa1bf53dd6e4ca954d47caebca4  1.7s
     => => resolve docker.io/library/ubuntu:22.04@sha256:4b1d0c4a2d2aaf63b37111f34eb9fa89fa1bf53dd6e4ca954d47caebca400  0.0s
     => => sha256:4b1d0c4a2d2aaf63b37111f34eb9fa89fa1bf53dd6e4ca954d47caebca4005c2 1.42kB / 1.42kB                      0.0s
     => => sha256:817cfe4672284dcbfee885b1a66094fd907630d610cab329114d036716be49ba 529B / 529B                          0.0s
     => => sha256:a8780b506fa4eeb1d0779a3c92c8d5d3e6a656c758135f62826768da458b5235 1.46kB / 1.46kB                      0.0s
     => => sha256:e96e057aae67380a4ddb16c337c5c3669d97fdff69ec537f02aa2cc30d814281 30.43MB / 30.43MB                    0.6s
     => => extracting sha256:e96e057aae67380a4ddb16c337c5c3669d97fdff69ec537f02aa2cc30d814281                           1.0s
     => [ 2/12] RUN apt-get update                                                                                      3.1s
     => [ 3/12] RUN apt-get install -y curl git wget unzip libglu1-mesa                                                 4.1s
     => [ 4/12] RUN apt-get clean                                                                                       0.2s
     => [ 5/12] RUN git clone https://github.com/flutter/flutter.git /usr/local/flutter                                11.8s
     => [ 6/12] RUN flutter doctor                                                                                     23.9s
     => [ 7/12] RUN flutter channel stable                                                                              1.5s
     => [ 8/12] RUN flutter upgrade                                                                                    18.6s
     => [ 9/12] RUN mkdir /app/                                                                                         0.4s
     => [10/12] COPY .. /app/                                                                                           0.0s
     => [11/12] WORKDIR /app/                                                                                           0.0s
     => [12/12] RUN ["chmod", "+x", "/app/docker/run_tests.sh"]                                                         0.2s
     => exporting to image                                                                                              2.9s
     => => exporting layers                                                                                             2.9s
     => => writing image sha256:0d09e4c38d1c08d536dd59be4ec4b87ce8e6da924ef6fbebbd75fc12f9e93354                        0.0s
     => => naming to docker.io/library/polkadart-tests                                                                  0.0s
    [+] Running 2/0
     ⠿ Network polkadart_default    Created                                                                             0.0s
     ⠿ Container polkadart-tests-1  Created                                                                             0.0s
    Attaching to polkadart-tests-1
    polkadart-tests-1  | /usr/bin/env: 'sh\r': No such file or directory
    polkadart-tests-1 exited with code 127
    
  • Could you add some documentation to the README, such as a brief intro to the project or just link to where the documentation can be found?
  • Could you provide a test suite for the metadata parser and encoding/decoding metadata? I could only find this test that checks the decoder doesn't throw an exception in one case.

@leonardocustodio
Copy link
Contributor

Hello @alxs, I made a change, can you give another go? If it doesn't work again can you please share with me the current environment that you are running? linux distro? windows? osx?

@alxs
Copy link
Contributor

alxs commented Nov 28, 2022

Hey @leonardocustodio, of course. It didn't work again, this is a VPS running Ubuntu 20.04.

[+] Running 1/0
 ⠿ Container polkadart-tests-1  Created                                                                             0.0s
Attaching to polkadart-tests-1
polkadart-tests-1  | /usr/bin/env: 'sh\r': No such file or directory
polkadart-tests-1 exited with code 127

@leonardocustodio
Copy link
Contributor

leonardocustodio commented Nov 28, 2022

Odd, I've removed the /usr/bin/env mention in the last PR could it be trying to run the cached image? After getting the latest commit on master could you try docker compose build --no-cache && docker compose up?

@alxs
Copy link
Contributor

alxs commented Nov 28, 2022

Sure! I don't get exactly the same error, but a very similar one:

docker compose build --no-cache && docker compose up

[+] Building 72.6s (17/17) FINISHED
 => [internal] load build definition from Dockerfile                                                                0.0s
 => => transferring dockerfile: 32B                                                                                 0.0s
 => [internal] load .dockerignore                                                                                   0.0s
 => => transferring context: 2B                                                                                     0.0s
 => [internal] load metadata for docker.io/library/ubuntu:22.04                                                     0.7s
 => CACHED [ 1/12] FROM docker.io/library/ubuntu:22.04@sha256:4b1d0c4a2d2aaf63b37111f34eb9fa89fa1bf53dd6e4ca954d47  0.0s
 => [internal] load build context                                                                                   0.0s
 => => transferring context: 67.01kB                                                                                0.0s
 => [ 2/12] RUN apt-get update                                                                                      3.0s
 => [ 3/12] RUN apt-get install -y curl git wget unzip libglu1-mesa                                                 3.6s
 => [ 4/12] RUN apt-get clean                                                                                       0.3s
 => [ 5/12] RUN git clone https://github.com/flutter/flutter.git /usr/local/flutter                                16.5s
 => [ 6/12] RUN flutter doctor                                                                                     25.6s
 => [ 7/12] RUN flutter channel stable                                                                              1.5s
 => [ 8/12] RUN flutter upgrade                                                                                    18.0s
 => [ 9/12] RUN mkdir /app/                                                                                         0.2s
 => [10/12] COPY .. /app/                                                                                           0.0s
 => [11/12] WORKDIR /app/                                                                                           0.0s
 => [12/12] RUN ["chmod", "+x", "/app/docker/run_tests.sh"]                                                         0.2s
 => exporting to image                                                                                              2.9s
 => => exporting layers                                                                                             2.9s
 => => writing image sha256:51184ef5860bd1fae46ee153bdec65554f12dc2c41e19f619e7c179f8bad6529                        0.0s
 => => naming to docker.io/library/polkadart-tests                                                                  0.0s
[+] Running 1/0
 ⠿ Container polkadart-tests-1  Recreated                                                                           0.0s
Attaching to polkadart-tests-1
polkadart-tests-1  | exec /app/docker/run_tests.sh: no such file or directory
polkadart-tests-1 exited with code 1

@leonardocustodio
Copy link
Contributor

leonardocustodio commented Nov 28, 2022

This one provides a bit more information because of the path. I believe I now know what is happening. No one from our team had this issue, even though Docker is supposed to run almost equally across everybody sometimes we do see differences. I will go for another fix and let you know when it is done, thank you very much

@leonardocustodio
Copy link
Contributor

Hey @alxs, can we try one more time? I was also wondering if this could be a CRLF issue. But you and I are both using Ubuntu so it shouldn't be that. If you can clone it again, the latest commit and make a fresh build with no cache. Thank you very much :)

@alxs
Copy link
Contributor

alxs commented Dec 1, 2022

@leonardocustodio I'm sorry I don't bring better news, but it is still failing:

[+] Building 74.1s (16/16) FINISHED
 => [internal] load build definition from Dockerfile                                                                0.0s
 => => transferring dockerfile: 543B                                                                                0.0s
 => [internal] load .dockerignore                                                                                   0.0s
 => => transferring context: 2B                                                                                     0.0s
 => [internal] load metadata for docker.io/library/ubuntu:bionic@sha256:eb1392bbdde63147bc2b4ff1a4053dcfe6d15e4dfd  0.0s
 => [internal] load build context                                                                                   0.0s
 => => transferring context: 353.89kB                                                                               0.0s
 => [ 1/11] FROM docker.io/library/ubuntu:bionic@sha256:eb1392bbdde63147bc2b4ff1a4053dcfe6d15e4dfd3cce29e9b9f52a4f  0.0s
 => [ 2/11] RUN apt-get update                                                                                      2.3s
 => [ 3/11] RUN apt-get install -y curl git wget unzip libglu1-mesa                                                10.3s
 => [ 4/11] RUN apt-get clean                                                                                       0.3s
 => [ 5/11] RUN git clone https://github.com/flutter/flutter.git /usr/local/flutter                                12.7s
 => [ 6/11] RUN flutter doctor                                                                                     24.9s
 => [ 7/11] RUN flutter channel stable                                                                              1.6s
 => [ 8/11] RUN flutter upgrade                                                                                    18.5s
 => [ 9/11] RUN mkdir /app/                                                                                         0.3s
 => [10/11] COPY .. /app/                                                                                           0.0s
 => [11/11] WORKDIR /app/                                                                                           0.0s
 => exporting to image                                                                                              3.1s
 => => exporting layers                                                                                             3.1s
 => => writing image sha256:3d1586d70ae7901e9da3ce0ffd287410d6787b2f8512da799edcb54f9aae01e9                        0.0s
 => => naming to docker.io/library/polkadart-tests                                                                  0.0s
[+] Running 1/0
 ⠿ Container polkadart-tests-1  Recreated                                                                           0.0s
Attaching to polkadart-tests-1
polkadart-tests-1  | exec ./run_tests.sh: no such file or directory
polkadart-tests-1 exited with code 1

That is on the latest main running the command you provided.

@leonardocustodio
Copy link
Contributor

leonardocustodio commented Dec 1, 2022

I'm starting to feel embarrassed... I removed this file. PR. If we still get an error. We will have to find a way to replicate it somehow. Can we give it another shot? Really sorry about making you do this. We are six people, and we have not managed to replicate this yet.

@alxs
Copy link
Contributor

alxs commented Dec 1, 2022

That did the trick! No worries at all. Let me know when I should take another look at the delivery.

@Lohann
Copy link
Contributor Author

Lohann commented Dec 6, 2022

@alxs ready for a second look 👀

@alxs
Copy link
Contributor

alxs commented Dec 9, 2022

@Lohann many thanks for the additional work and also for adding support for pre-v14 metadata, which wasn't even in the scope of this milestone!

After reviewing the code and your latest commits, it appears that you've implemented the encoding/decoding of some of the metadata contents as you list them here, but not of all of them. To my understanding these will be necessary to construct transactions, but there's nothing else in your milestone plan between here and milestones 2 (networking) and 3 (cryptography). Am I missing something?

Otherwise, could you ensure that this is fully implemented before moving on to the next milestone? You can always amend the milestone plan if you want to redistribute the work or adjust the budget.

@Lohann
Copy link
Contributor Author

Lohann commented Dec 9, 2022

Hi @alxs I think there's a confusion about what metadata means, once it have two meanings depending of the context, one is frame-metadata, which is the format on how frame represents it's internal types, what we did in the milestone 1 is automatically parse all types from frame-metadata.
Example how to retrieve the frame-metadata from polkadot:

curl -H "Content-Type: application/json" -d '{"id":1,"jsonrpc":"2.0","method":"state_getMetadata","params":[]}' https://rpc.polkadot.io

Parse frame-metadata is a BIG milestone, because it requires the dart scale-codec (which deserved a grant by itself), ss58 for retrieve the address public key, just then is possible to parse the frame-metadata, which is not trivial as you can check here.

The metadata described here is another thing, it is the interface in Dart that allows the user to interact with the node, which includes read/mutate the chain state, similar to what polkadot-api does here:
https://polkadot.js.org/docs/substrate

Notice that both are called metadata but are different things, before the Milestone 2 is not possible to implement JSON-RPC, Constants, Read Storage etc, because it requires the RPC and websockets layer to be implemented.

To clarify, I'll send an example of what you should expect to be delivered in each milestone:

Milestone 1

Goal: Encode/Decode scale-codec, ss58 addresses and parse frame-metadata

Scale Codec ✅

// defining used types
final typesRegistry = <String, dynamic>{
  'Codec': {
    'd': 'Compact<u32>',
  },
};

// Creates the registry for parsing the types and selecting particular schema.
final registry = TypeRegistry(types: typesRegistry);

// specifyng which schema type to use
registry.select('Codec');

// fetching the parsed types from `Json` to `Type`
final types = registry.getTypes();

// Initializing Scale-Codec object
final codec = Codec(types);

// encoding and decoding 16777215 as Compact<u32>
var encodedInt = codec.encode(registry.getIndex('Compact<u32>'), 16777215);
var decodedInt = codec.decode(registry.getIndex('Compact<u32>'), encodedInt);

More examples here

SS58 ✅

// get registry info of given `prefix`
final polkadotRegistry = Codec.registry.getByPrefix(0);

// decoding substrate address
final String originalAddress ='5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY';
final List<int> publicKey = polkadotRegistry.decode(originalAddress);

More examples here

Parse frame-metadata ✅

final decoder = MetadataDecoder();

// Really big hexadecimal string
// curl -H "Content-Type: application/json" -d '{"id":1,"jsonrpc":"2.0","method":"state_getMetadata","params":[]}' https://rpc.polkadot.io
final metadata_hex = "0x6d6574610e6908000c1c....7461"
var decodedMetadata = decoder.decode(metadata_hex);

Example here
Obs: notice this milestone doesn't cover how to retrieve the metadata, only how to parse it, retrieve metadata requires JSON-RPC which will be done in Milestone 2

Milestone 2 🟡

Goal: Expose read-only methods

var api = await PolkadartApi.connect("wss://rpc.polkadot.io");

// RPC
var blockHash = await api.rpc.chain.getBlockHash(1)

// Constants
var ed = await api.consts.balances.existentialDeposit()
 
// Storage
var balance = await api.query.balances.account(address)

Milestone 3 🔴

Goal: Create/manage accounts, sign and send extrinsics.
Sign extrinsics, derive wallet, etc, require Keyring and Crypto libraries.

// create a keyring for ALICE account
var keyring = new Keyring({ type: 'sr25519', ss58Format: 2 });
var mneumonic = "floor author slight rebuild network naive spirit climb income stool brief clay net yard document";
var alice = keyring.addFromUri(mnemonic, "ed25519");

// Connect to WS API
var api = await PolkadartApi.connect("wss://rpc.polkadot.io");

// Create transaction
var extrinsic = await api.tx.balances.transferKeepAlive(dest, amount);

// Alice sign and send transaction
await extrinsic.signAndSend(alice)

Obs: the code above is just an example of which features will be delivered in each milestone, the interfaces and the final code may be different.

@alxs
Copy link
Contributor

alxs commented Dec 14, 2022

@Lohann we may be talking past each other here. What I meant to say is, isn't the reason that the frame-metadata is provided so one can communicate with the node? You don't only want to parse it correctly, you want to be able to understand and represent all types and other information provided in the metadata so that you can also create and encode extrinsics, RCP calls etc. Even if you don't have JSON-RPC and can't sign transactions, you'll still want to be able to create and encode them. Am I missing something?

@Lohann
Copy link
Contributor Author

Lohann commented Dec 14, 2022

@alxs

isn't the reason that the frame-metadata is provided so one can communicate with the node?

Currently we don't have a way to communicate with Substrate node, once it requires a websocket connection between polkadart and the node, and this will be done in Milestone 2. Frame-metadata scope doesn't include how to communicate to the substrate node, only how to encode/decode it's metadata.

Even if you don't have JSON-RPC and can't sign transactions, you'll still want to be able to create and encode them.

Notice that is already possible to encode/decode extrinsics, constants and RPC calls, etc using Polkadart Metadata, as you can see here:
https://github.com/rankanizer/polkadart/tree/main/packages/substrate_metadata#decode-extrinsic

Each metadata version expose different types depending of the version:
https://github.com/rankanizer/polkadart/blob/main/packages/substrate_metadata/lib/models/metadata/metadata_versions.dart#L65-L93

So the Metadata is ready, it already encode/decode extrinsics, storage, constants, but this is only useful once we connect to the node, so we can effectively use this to read the chain state.

@alxs
Copy link
Contributor

alxs commented Dec 20, 2022

Thanks for the explanation and for adding more examples to the documentation. That makes sense - I was able to find the functionality thanks to the new examples and a bit more digging and, as I understand it now, this list represents the functionality of the library and is otherwise unrelated to Substrate metadata. Perhaps renaming this section would prevent confusion down the line.

And with this, I'm happy to let you know that the milestone has been accepted! Thank you for the great work on this milestone. You can find my evaluation notes here.

I'll notify the operations team to pay out the invoice you submitted. Please allow for up to 14 days for processing.

@alxs alxs merged commit 9cc80a9 into w3f:master Dec 20, 2022
@github-actions
Copy link

Congratulations on completing the first milestone of this grant! As part of the Grants Program, we want to help grant recipients acknowledge their grants publicly. To that end, we’ve created a badge for projects that successfully deliver their first milestone. Note that it must only be used within the context of the delivered work, so please do not display it on your team or project's homepage unless accompanied by a short description of the grant.

Furthermore, you're now welcome to announce the grant publicly. Please remember to observe the foundation’s guidelines in doing so. In case you haven't done so yet, you may also reach out to grantsPR@web3.foundation for feedback on your announcement and cross-promotion.

Thank you for your contribution and good luck with the remaining milestones, if any! As usual, please let us know if you run into any delays by leaving a comment on the application PR, or directly submitting an amendment.

@alxs
Copy link
Contributor

alxs commented Dec 20, 2022

@Lohann I just noticed that your invoice is in USDT, but "only" USDC, USD and DOT are mentioned in the application :)

Could you submit a quick amendment replacing DOT with USDT here, and USDC with USD here? Thank you.

@alxs
Copy link
Contributor

alxs commented Dec 20, 2022

Also please update the address with the one in your invoice.

@Lohann
Copy link
Contributor Author

Lohann commented Dec 20, 2022

@alxs will do!

@Lohann
Copy link
Contributor Author

Lohann commented Dec 20, 2022

I put USDT because that's the only stablecoin currently supported by statemint:
https://tether.to/en/tether-usdt-to-launch-on-polkadot/

@alxs
Copy link
Contributor

alxs commented Dec 21, 2022

@Lohann we try to ensure the address and currency in the invoice are always the same ones as in the application.

@alxs
Copy link
Contributor

alxs commented Dec 29, 2022

@Lohann friendly reminder

@alxs
Copy link
Contributor

alxs commented Dec 30, 2022

Please ping @keeganquigley when you've submitted the amendment and it's been approved and he will forward the invoice to the operations team.

@keeganquigley keeganquigley self-assigned this Jan 3, 2023
@Lohann
Copy link
Contributor Author

Lohann commented Jan 4, 2023

@alxs @keeganquigley Invoice updated, it can be paid in DOT instead USDT.

@keeganquigley
Copy link
Contributor

Thanks @Lohann however, we don't offer payments in DOT. Please see the available payment methods.

I believe Alex was not asking you to update the invoice, but rather submit an amendment to update your application to remove DOT and switch it to USDT here.

Total costs should be displayed in US Dollars, therefore in addition he asked you to also remove the USDC reference here, by changing it to USD instead.

I hope that makes sense and sorry for the confusion. Your original invoice should be fine. The amendment will need 5 approvals from the committee (the same as your original application) but these approvals should happen relatively quickly. Thanks and I will watch for it!

@Lohann
Copy link
Contributor Author

Lohann commented Jan 6, 2023

@keeganquigley Hmm so DOT is not support anymore, ok amend PR created!
w3f/Grants-Program#1416

@keeganquigley
Copy link
Contributor

@Lohann that's correct thank you! Please submit a new invoice accordingly since it is now USDC.

@Lohann
Copy link
Contributor Author

Lohann commented Jan 6, 2023

@keeganquigley done ✅

@keeganquigley
Copy link
Contributor

Thank you @Lohann I have forwarded your invoice for payment. Please expect up to 14 days for delivery.

@RouvenP
Copy link

RouvenP commented Jan 20, 2023

hi @Lohann we just transferred the payment. Apologies for the delay!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants