Metadata

[JoshOrndorff]

We should consider using metadata so we can have a dynamic wallet to submit transactions. We could take inspiration from https://github.com/paritytech/frame-metadata.

[JoshOrndorff]

@NadigerAmit Please post your findings and thoughts here as you learn.

[NadigerAmit]

Current Metadata situation of Tuxedo
Tuxedo doesn’t support the metadata implementation as of now.
Hence Polkadot.js can’t show the blockchain data and it is difficult for engineers to test the runtime.

Why metadata is required.
Interoperability: Metadata enables external applications to understand and interact with the blockchain's structure and functionality.

User Interfaces: Metadata provides essential information for constructing user-friendly interfaces in wallets and explorers.
Automatic UI Generation: Metadata supports automatic UI generation, helping tools like Polkadot.js dynamically create transaction forms.

Documentation: Metadata serves as self-documentation for the blockchain's runtime, describing types, storage, and transactions.

Upgradability: Metadata supports runtime upgradability, allowing descriptions of changes in the on-chain runtime state.

Developer Tooling: Developers use metadata to understand runtime capabilities, design smart contracts, and interact during development.

Cross-Chain Communication: In multi-chain environments, metadata facilitates communication and interaction between different blockchains.

Why Tuxedo don’t have support for Metadata

Tuxedo is also built based on the substrate
BUT
Tuxedo doesn’t use FRAME(FRAME is account-based) as it is UTXO-based.
Metadata data is mainly implemented in FRAME (below modules)
Frame_metadadata crate
Frame_system
Frame_support
Construct_runtime
polkadot-sdk/substrate/frame/support/procedural/src/construct_runtime/expand/metadata.rs
polkadot-sdk/substrate/frame/support/procedural/src/construct_runtime/mod.rs
https://github.com/paritytech/cargo-contract/blob/3db4734c1a056f8834f0b0e35a77a4ef334fc79f/src/cmd/metadata.rs

Requirements of metadata for Tuxedo

Dynamic Metadata Generation: The framework must support dynamic generation of metadata for runtime modules (Pieces) during initialization.
Justification: This allows new runtime modules to register their metadata seamlessly without requiring manual intervention.

Metadata Structure: The framework must define a structured metadata format, including information about storage entries, supported transactions, constraint checker errors, Required constraints on the input and output, and Required Verifiers for transactions such as sign, multisign, upForGrab,etc.

Dynamic Extrinsic Metadata: The framework must allow runtime modules to dynamically register metadata for extrinsics.
Justification: This supports dynamic registration of information about supported extrinsics.

Extrinsic Metadata Structure: The metadata structure must include information about supported extrinsic.
Justification: External applications such as polkdot.js need to understand the structure and parameters of available extrinsic.

Registration Mechanism: The framework must provide a registration mechanism (macro or trait) for runtime modules to register their metadata.
Justification: A standardized registration process ensures uniformity across different runtime modules.

Serialization and Deserialization The framework must support serialization and deserialization of metadata using the SCALE codec library.
Justification: Serialization enables efficient storage and transmission of metadata, while deserialization allows external applications to interpret the information.

Integration with SCALE Codec: The framework must integrate with the SCALE codec library, utilizing the scale-info crate for generating metadata.
Justification: Integration with SCALE ensures compatibility with Substrate-based blockchains and enables consistent encoding and decoding of data.

Polkadot.js API Compatibility: The framework must expose an API endpoint compatible with Polkadot.js (state_getMetadata()).
Justification: Polkadot.js relies on metadata to understand the state and functionality of the blockchain, and a compatible API facilitates seamless interaction.

Polkadot.js UI Integration: The framework must support integration with the Polkadot.js UI, dynamically displaying information about registered Pieces.
Justification: This enables users to interact with the blockchain through a user-friendly interface.

Opaque Data Structure: The metadata structures must be designed as opaque data structures to abstract internal details from external applications.
Justification: Opaque structures enhance flexibility, maintainability, and versioning of the metadata.

[NadigerAmit]

Below is 1st Draft Design :
We want to follow a design approach similar to FRAME and leverage the existing RPC endpoint and Runtime API for providing metadata in Tuxedo, We can structure our metadata design accordingly.

The important thing to notice is that FRAME is account-based and Tuxedo is UTXO-based. The tuxedo doesn’t have below :

1. Storage
2. Function Call
3. Events

Instead, Tuxedo has below :

1. Inputs
2. Outputs
3. Peeks
4. Constraint checkers
5. Verifiers
6. Constraints errors.

Below is data structure I am thinking of now

`#[derive(TypeInfo)]
pub struct PieceMetadata {  
    pub constraint_checker_errors: Vec<ConstraintCheckerError>,   
    pub extrinsics: Vec<ExtrinsicEntry>, 
     pub documentation: Vec<String>,
}

#[derive(TypeInfo)] 
pub struct ParameterEntry {
    pub name: String,
    pub modifier: ParamEntryModifier, 
    pub ty: DataType, 
    pub default: Vec<u8>, 
    pub documentation: Vec<String>,
}

#[derive(TypeInfo)]
pub enum ConstraintCheckerError {
    // Define constraint checker errors
    // Example: NotEnoughFunds, InvalidInput, etc.
}

#[derive(TypeInfo)]
pub struct ExtrinsicEntry {
    pub name: String,
    pub inputs: Vec<ParameterEntry>,
    pub peeks: Vec<ParameterEntry>,
    pub outputs: Vec<ParameterEntry>,
    pub constraintCheckerType:FreeKittyConstraintChecker
    pub documentation: Vec<String>,
}

pub enum ConstraintChecker {
    // Define supported transaction types such as below
    Mint, 
    Spend,
    Transfer,
    Burn,
}

#[derive(TypeInfo)] 
pub enum DataType { 
    Int, 
    Bool, 
    Array,
    // we can add more 
}
`

Draft Algorithm:Steps to generate the metadata:

Step 1: Define Piece Metadata Trait
The PieceMetadataTrait defines a method metadata() that should be implemented by each piece to provide its metadata.
This trait needs to be implemented by the different pieces. So that they will provide the metadata to the central module.

pub trait PieceMetadataTrait {
    fn metadata() -> PieceMetadata;
}

Step 2: Implementation of PieceMetadataTrait for Piece.

Implementation of PieceMetadataTrait
for Piece.

The KittyPiece struct implements
PieceMetadataTrait, and we should adjust
the metadata in the metadata() method
based on the actual extrinsics, errors,
and documentation.

impl PieceMetadataTrait for KittyPiece {
    fn metadata() -> PieceMetadata {
        PieceMetadata {
            constraint_checker_errors: vec![
                ConstraintCheckerError::BadlyTyped,
                // Add other errors...
            ],
            extrinsics: vec![
                ExtrinsicEntry {
                    name: "Breed".to_string(),
                    inputs: vec![
                        ParameterEntry {
                            name: "mom".to_string(),
                            modifier: ParamEntryModifier::None, // Add the actual modifier
                            ty: DataType::Array, // Add the actual data type
                            default: vec![],
                            documentation: vec!["Input parameter for Breed extrinsic".to_string()],
                        },
                        // Add other input parameters such as dad ...
                    ],
                    peeks: vec![
                        // Add peek parameters...
                    ],
                    outputs: vec![
                        // Add output parameters...
                    ],
                    constraint_checker_type: FreeKittyConstraintChecker::Breed,
                    documentation: vec!["Breed extrinsic documentation".to_string()],
                },
                // Add other extrinsics...
            ],
            documentation: vec![
                "Documentation for Piece".to_string(),
                // Add more documentation...
            ],
        }
    }
}

Note: Currently generation of metadata by individual runtime pieces is done manually i.e. piece developer needs to implement the metadata() function from the PieceMetadata trait.
I think we need to define the macro as the final stage which can generate the code for doing this automatically.

Step 3: Centralized Module for Metadata Collection and Registration.
The MetadataRegistry is a centralized module that collects and registers metadata for all registered pieces.
The register_piece method is used to add the metadata of a piece to the registry, and encode_metadata encodes the collected metadata using the SCALE codec.

use parity_scale_codec::Encode;
use scale_info::TypeInfo;

pub struct MetadataRegistry {
    pub pieces_metadata: Vec<PieceMetadata>,
}

impl MetadataRegistry {
    pub fn new() -> Self {
        MetadataRegistry {
            pieces_metadata: Vec::new(),
        }
    }

    pub fn register_piece<T: PieceMetadataTrait + TypeInfo>(&mut self) {
        let metadata = T::metadata();
        self.pieces_metadata.push(metadata);
    }

    pub fn encode_metadata(&self) -> Vec<u8> {
        self.pieces_metadata.encode()
    }
}

Step 4: Retrieve the metadata in the from the runtime.

/// Example custom runtime module
pub struct TuxedoRuntime;

impl TuxedoRuntime {
    pub fn get_metadata() -> Vec<u8> {
        let metadata = MetadataRegistry::new().encode_metadata();
        metadata
    }
}

// Implement  own trait for fetching metadata in runtime
pub trait MetadataProvider {
    fn get_metadata() -> Vec<u8>;
}

impl MetadataProvider for TuxedoRuntime {
    fn get_metadata() -> Vec<u8> {
        MyRuntime::get_metadata()
    }
}

Step 5: Provide Metadata via state_getMetadata()

Provide Metadata via state_getMetadata()

From the sp_api APIs

impl sp_api::Metadata<Block> for Runtime {
    fn metadata() -> OpaqueMetadata {
        let metadata = TuxedoRuntime::get_metadata();
        OpaqueMetadata::new(metadata)
    }

    fn metadata_at_version(_version: u32) -> Option<OpaqueMetadata> {
        // We might implement versioning logic here if needed
        None
    }

    fn metadata_versions() -> Vec<u32> {
        // We might implement versioning logic here if needed
        Default::default()

[NadigerAmit]

OpaqueMetadata in Substrate refers to encoding metadata into a format that is not directly human-readable, often done for efficiency or security reasons. This encoded metadata can be retrieved by clients like Polkadot.js and used to understand the runtime's capabilities and structures.
Stored in the sp_core lib.rs - source (docs.rs) as below :

Step1: Metadata is defined in the sp_core (lib.rs - source (docs.rs) )

/// Stores the encoded `RuntimeMetadata` for the native side as an opaque type.
#[derive(Encode, Decode, PartialEq, TypeInfo)]
pub struct OpaqueMetadata(Vec<u8>);

Step2: Definition the metadata trait in the sp_api
Metadata trait is defined in the sp_api(lib.rs - source (paritytech.github.io)) as below :

decl_runtime_apis! {
	/// The `Core` runtime api that every Substrate runtime needs to implement.
                                         #[core_trait]
	#[api_version(4)]
	pub trait Core {
		/// Returns the version of the runtime.
		fn version() -> RuntimeVersion;
		/// Execute the given block.
		fn execute_block(block: Block);
		/// Initialize a block with the given header.
		#[renamed("initialise_block", 2)]
		fn initialize_block(header: &<Block as BlockT>::Header);
	}
           
/// The `Metadata` API trait that returns metadata for the runtime.
	#[api_version(2)]
	pub trait Metadata {
		/// Returns the metadata of a runtime.
		fn metadata() -> OpaqueMetadata;

		/// Returns the metadata at a given version.
		///
		/// If the given `version` isn't supported, this will return `None`.
		/// Use [`Self::metadata_versions`] to find out about supported metadata version of the runtime.
		fn metadata_at_version(version: u32) -> Option<OpaqueMetadata>;

		/// Returns the supported metadata versions.
		///
		/// This can be used to call `metadata_at_version`.
		fn metadata_versions() -> sp_std::vec::Vec<u32>;
	}
}

Step 3: Implementing the metadataTrait in Runtime

Implement the metadata trait in Runtime

Inside the implementation of the Metadata trait, we encode runtime's metadata into a Vec using your encoding logic.

The goal is to represent all the necessary information about runtime in a compact, binary form.

 // Tuxedo does not yet support metadata
impl sp_api::Metadata<Block> for Runtime {
    fn metadata() -> OpaqueMetadata {
        OpaqueMetadata::new(Runtime::metadata().into())
    }

    fn metadata_at_version(version: u32) -> Option<OpaqueMetadata> {
        Runtime::metadata_at_version(version)
    }

    fn metadata_versions() -> sp_std::vec::Vec<u32> {
        Runtime::metadata_versions()
    }
}

Step 4: Retrieving Metadata
Clients like Polkadot.js can then use the state_getMetadata RPC call to retrieve this opaque metadata from the Substrate node. The encoded data can be interpreted by the client to understand the runtime's structures, supported extrinsics, and other relevant information.

For example, in Polkadot.js, we need to call api.rpc.state.getMetadata() to get the encoded metadata.

Step 5: Client Interpretations of Metadata

Once the encoded metadata is retrieved, Polkadot.js or other clients need to have logic to decode this opaque format and display it in a more human-readable and user-friendly manner.
This decoding process allows users and developers to understand the runtime's features and capabilities.

[JoshOrndorff]

A couple little thoughts based on what you wrote:

• scale, type-info, serde You're exactly on the right track with this. We will use all three. The type info is one of the most important parts, and I'm hoping we can copy it directly from FRAME.
• Polkadot.js Don't get too excited about polkadot js at first. It assumes it is working with FRAEM is a LOT of places. Our first goal is to make the existing CLI wallet, and your upcoming browser dApp work dynamically. Once we've done that, then we can try to get some parts of Polkadot js working.
• Recursive aggregation In Tuxedo a piece can be aggregated from smaller constituent pieces, which can themselves be aggregated from other smaller pieces. This is different than frame and our metadata format will need to support it.

[JoshOrndorff]

Made a tiny bit of progress on this in #211