PE-4318: implement stream functions #52

karlprieb · 2023-08-23T15:40:02Z

No description provided.

lib/src/streams/data_bundle.dart

lib/src/streams/data_item.dart

atticusofsparta · 2023-08-31T18:05:33Z

lib/src/streams/data_item.dart

+      input: signatureData,
+      signature: signature,
+      modulus: decodeBytesToBigInt(owner),
+      publicExponent: BigInt.from(65537));


It seems to be the Largest known Fermat prime. We could give it a name if we make a constant for this value.

0077047 (#52)

dtfiedler · 2023-09-01T15:08:54Z

there's likely similar patterns for handling streams and packaging bundles relevant to the turbo-sdk ardriveapp/turbo-sdk#9

matibat

Reviewed half of the code. I'll continue tomorrow. Looking great so far. 🚀

.gitignore

matibat · 2023-09-18T17:12:04Z

flake.nix

@@ -0,0 +1,20 @@
+{
+  description = "A basic shell for arweave-dart";
+  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";


Do we need unstable packages?

It's only for nix env (which I'm the only on using right now).
nixpkgs has 3 types of releases, the master branch which is unstable, unstable branch which is like a rolling release RC and tagged branches which are stable. Stable tags are updated at each 6˜12 months.
Flutter and dart v3 are not on tagged branches yet.
unstable doesn't mean it's really unstable, just that it was not tagged yet.

matibat · 2023-09-18T17:32:33Z

js/sha384/dist/sha384.js

Don't we want to ignore this file?

Perhaps the whole folder, I'd say

I don't think so. It's expected for users to copy and use this js file if they want to support web environments

I mean, this is generated code and not source files. We usually git-ignore the generated code because we do generate it on build time.

But if we're aware of that and this is intended then I'm good with it.

js/sha384/package.json

js/sha384/src/index.ts

matibat · 2023-09-18T19:32:18Z

lib/src/streams/data_item.dart

+  final tagsBytes = decodeBytesToLong(await reader.readBytes(8));
+  byteIndex += 8;


This one seems to be an unused variable. Is it perhaps a duplicate of what's below?

It's unused, but not duplicated. The first 8 bytes determine the bytes of the tags and we don't use it for anything right now, but we should read them from the stream as we follow the spec order of bytes

Ya, but isn't it the same as what's below? If not then let's consider using another name to differentiate both.

// get tags bytes length final tagsBytesLenthBytes = await reader.readBytes(8); final tagsBytesLength = decodeBytesToLong(tagsBytesLenthBytes); byteIndex += 8;

We just confirmed with @thiagocarvalhodev that the first one represents the amount of tags there are, and the second one represents the total size in bytes for the whole tags altogether.

We could rename them to make obvious the difference.

matibat · 2023-09-18T19:47:29Z

lib/src/streams/data_item.dart

+      input: signatureData,
+      signature: signature,
+      modulus: decodeBytesToBigInt(owner),
+      publicExponent: BigInt.from(65537));


It seems to be the Largest known Fermat prime. We could give it a name if we make a constant for this value.

lib/src/streams/data_models.dart

matibat · 2023-09-18T19:54:36Z

lib/src/streams/deep_hash_stream.dart

+  }
+}
+
+Future<Uint8List> deepHashChunks(dynamic chunks, Uint8List acc) async {


Should we type chunks? It seems to be a list-like object.

deepHashChunks can be either a Stream<Uint8List>, Uint8List, a List<List> or List<Stream<Uint8List>>. As Dart doesn't have union types, using dynamic was the best option that I found :(

Do you have any suggestion?

I was thinking of making the type be Iterable<dynamic> but I see Stream implements neither List nor Iterable.

What we can do is what you did for deepHashStream: to check if data is Stream<Uint8List> or data is List<List> || data is List<Stream<Uint8List>>, and to throw otherwise.

Having the method as it is (accessing properties we assume it has) is dangerous. 😬

lib/src/streams/sha384_web.dart

matibat

Finished reviewing :)

matibat · 2023-09-20T18:40:22Z

lib/src/streams/utils.dart

+        input: signatureData,
+        signature: signature,
+        modulus: decodeBytesToBigInt(owner),
+        publicExponent: BigInt.from(65537));


We're using this number in many places, let's make a constant for it. Or perhaps to factor out this code into it's own method: bool isSignatureValid(signatureData, signature, owner)

lib/src/streams/utils.dart

pubspec.lock

test/data_bundle_test.dart

matibat · 2023-09-20T19:09:13Z

test/data_item_test.dart

+  group('[streams][data_item]', () {
+    test('create and sign data item', () async {


Nit: we usually set the top-level group's description as the name of the component we're testing so you can read: "Component X behaves like THIS with the input A" - E.g. "createDataItemTaskEither method" "instantiates a stream that creates and sign a data item"

- use the url to the git

- implements the abort upload functionality. It adds to new classes: `ChunkUploader` and `UploadAborter`. The API for the method for upload also changed.

…-stop-file-upload-when-using-a-rs-only PE-4812: add option to cancel an ongoing upload

PE-3697: Download tx

- uses the fetch_client fork with the cancel option - bumps the package version

PE-4889: release v2.22.0

matibat

matibat · 2023-10-26T18:01:26Z

lib/src/streams/data_bundle.dart

+typedef BundledDataItemResult
+    = Future<TaskEither<StreamTransactionError, DataItemResult>>;
+BundledDataItemResult createBundledDataItemTaskEither({


It seems createDataBundleTaskEither is returning DataBundleResult instead of DataItemResult.

matibat · 2023-10-26T18:02:08Z

lib/src/streams/data_bundle.dart

+// Will be refactored to use TaskEither
+
+// Future<List<Map<String, dynamic>>> processBundle(


Something to do here? Can be cleaned up?

matibat · 2023-10-26T18:11:22Z

lib/src/streams/data_item.dart

+  final tagsBytes = decodeBytesToLong(await reader.readBytes(8));
+  byteIndex += 8;


We just confirmed with @thiagocarvalhodev that the first one represents the amount of tags there are, and the second one represents the total size in bytes for the whole tags altogether.

We could rename them to make obvious the difference.

matibat · 2023-10-26T18:22:34Z

lib/src/streams/http_client/browsers.dart

+
+Client getClient() => FetchClient(
+      mode: RequestMode.cors,
+      // streamRequests: true,


Commented code.

wip

72a66a3

karlprieb marked this pull request as draft August 23, 2023 15:40

update dart container

0d4a407

karlprieb force-pushed the PE-4318 branch 2 times, most recently from 21142d6 to 9f263f5 Compare August 25, 2023 20:45

update tests

a4bad99

karlprieb force-pushed the PE-4318 branch from 9f263f5 to a4bad99 Compare August 25, 2023 20:52

karlprieb added 2 commits August 25, 2023 17:57

add firefox to tests

d15c67e

remove tests on firefox for now

ea47f15

karlprieb force-pushed the PE-4318 branch from 9e715c3 to 3c1ecdd Compare August 25, 2023 21:21

skip import on dart vm

91f5ce8

karlprieb force-pushed the PE-4318 branch 2 times, most recently from 6bc1878 to 1fd8170 Compare August 30, 2023 17:36

karlprieb added 2 commits August 30, 2023 14:40

wip

3b14323

Refactor createDataItem and add createDataBundle

f653ed1

karlprieb force-pushed the PE-4318 branch from 1fd8170 to f653ed1 Compare August 30, 2023 17:40

use better name for props

59723b8

atticusofsparta reviewed Aug 31, 2023

View reviewed changes

lib/src/streams/data_bundle.dart Show resolved Hide resolved

atticusofsparta reviewed Aug 31, 2023

View reviewed changes

lib/src/streams/data_item.dart Show resolved Hide resolved

atticusofsparta reviewed Aug 31, 2023

View reviewed changes

karlprieb added 4 commits September 1, 2023 15:14

new createBundledDataItem implementation

0c7f139

WIP: testing different sha384

2c42b26

WIP: testing different sha384.

cf5c7e2

BDI generators on web are fast now

0cb7d21

karlprieb force-pushed the PE-4318 branch from 5627bfe to 0cb7d21 Compare September 12, 2023 13:06

add exports and remove map

7183b84

karlprieb marked this pull request as ready for review September 18, 2023 12:55

matibat reviewed Sep 18, 2023

View reviewed changes

matibat reviewed Sep 20, 2023

View reviewed changes

karlprieb and others added 10 commits September 21, 2023 15:41

.

a8d8e68

.

24fccb7

transaction uploader

ccd200c

Add stream downloader

1e4b57b

use fetch_client on browsers

2187852

fix: bundle generation for bdi

30517c2

use publicExponent instead of 65537 magic number

0077047

set dataItemHeaderSize

3a83992

add comments for transaction type and format

3a296fb

remove unused function _hexStringToUint8List

cb9fe81

karlprieb changed the title ~~PE-4318: implement stream functions [WIP]~~ PE-4318: implement stream functions Oct 6, 2023

karlprieb and others added 8 commits October 6, 2023 10:51

remove logs

4aa4377

Merge branch 'PE-4318' into PE-3697

cbd9670

Update pubspec.yaml

c2bd4e0

- use the url to the git

feat(upload aborter)

0fd8255

- implements the abort upload functionality. It adds to new classes: `ChunkUploader` and `UploadAborter`. The API for the method for upload also changed.

Merge pull request #54 from ardriveapp/PE-4812-cancel-button-fails-to…

05b7049

…-stop-file-upload-when-using-a-rs-only PE-4812: add option to cancel an ongoing upload

Merge pull request #53 from ardriveapp/PE-3697

e2c98c1

PE-3697: Download tx

Update pubspec.yaml

cf20380

- uses the fetch_client fork with the cancel option - bumps the package version

Merge pull request #55 from ardriveapp/PE-4889-release-v-2-22-0

9127d4a

PE-4889: release v2.22.0

thiagocarvalhodev self-assigned this Oct 26, 2023

lock fetch_api version

3d02630

thiagocarvalhodev approved these changes Oct 26, 2023

View reviewed changes

matibat approved these changes Oct 26, 2023

View reviewed changes

thiagocarvalhodev merged commit a61f9d7 into master Oct 26, 2023
3 checks passed

thiagocarvalhodev deleted the PE-4318 branch October 26, 2023 18:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PE-4318: implement stream functions #52

PE-4318: implement stream functions #52

karlprieb commented Aug 23, 2023

atticusofsparta Aug 31, 2023

matibat Sep 18, 2023

karlprieb Oct 6, 2023

dtfiedler commented Sep 1, 2023

matibat left a comment

matibat Sep 18, 2023

karlprieb Oct 6, 2023

matibat Sep 18, 2023

matibat Sep 18, 2023

karlprieb Oct 6, 2023

matibat Oct 10, 2023

matibat Sep 18, 2023

karlprieb Oct 6, 2023

matibat Oct 10, 2023

matibat Oct 26, 2023

matibat Sep 18, 2023

matibat Sep 18, 2023

karlprieb Oct 6, 2023

matibat Oct 10, 2023

matibat left a comment

matibat Sep 20, 2023

matibat Sep 20, 2023

matibat left a comment

matibat Oct 26, 2023

matibat Oct 26, 2023

matibat Oct 26, 2023

matibat Oct 26, 2023

		final tagsBytes = decodeBytesToLong(await reader.readBytes(8));
		byteIndex += 8;

		group('[streams][data_item]', () {
		test('create and sign data item', () async {

		// Will be refactored to use TaskEither

		// Future<List<Map<String, dynamic>>> processBundle(

PE-4318: implement stream functions #52

PE-4318: implement stream functions #52

Conversation

karlprieb commented Aug 23, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dtfiedler commented Sep 1, 2023

matibat left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matibat left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matibat left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment