feat: Data Submission Worker Integration. #51

heemankv · 2024-07-22T13:17:41Z

This PR resolves Issue #37.

Includes creation of multiple data submission jobs together.

Logic :

We assume absolute non sequential calls, no need to wait on previous data submission jobs to be completed before triggering for new block.
Y : Take last completed Proof Job.
X : Take last triggered Data Submission Job.
X <= Y
Trigger data submission jobs for all Y - X jobs.

crates/orchestrator/src/workers/data_submission.rs

…tring from VerificationFailed

crates/orchestrator/src/workers/snos.rs

apoorvsadana

We haven't added test cases for this code

CHANGELOG.md

crates/orchestrator/src/jobs/mod.rs

crates/orchestrator/src/workers/mod.rs

crates/orchestrator/src/workers/data_submission.rs

crates/orchestrator/src/workers/snos.rs

Co-authored-by: Apoorv Sadana <95699312+apoorvsadana@users.noreply.github.com>

…_by_type_and_status

crates/orchestrator/src/database/mongodb/mod.rs

crates/orchestrator/src/jobs/mod.rs

crates/orchestrator/src/workers/mod.rs

crates/orchestrator/src/workers/snos.rs

…s in verify_jobs

crates/orchestrator/src/database/mongodb/mod.rs

crates/orchestrator/src/jobs/mod.rs

Co-authored-by: Apoorv Sadana <95699312+apoorvsadana@users.noreply.github.com>

crates/orchestrator/src/workers/data_submission.rs

crates/orchestrator/src/workers/mod.rs

crates/orchestrator/src/workers/data_submission.rs

crates/orchestrator/src/workers/mod.rs

crates/orchestrator/src/workers/data_submission.rs

…bled, get_jobs_by_statuses

…Worker

* update: DA job draft #1 * docs: changelog updated * update: is_worker_enabled impl & usage in da_submission, removal of String from VerificationFailed * update: renamed to * update: run worker only if it's enabled using is_worker_enabled check * build: linter fixes * Update CHANGELOG.md Co-authored-by: Apoorv Sadana <95699312+apoorvsadana@users.noreply.github.com> * update: limit_to_one on get_jobs_by_status * update: removed get_last_successful_job_by_type, added get_latest_job_by_type_and_status * update: added error to job metadata * update: pr resolution, simplifying get_jobs_by_status, rejected status in verify_jobs * update: linting fixes * Update crates/orchestrator/src/jobs/mod.rs Co-authored-by: Apoorv Sadana <95699312+apoorvsadana@users.noreply.github.com> * update: removing .expect from mongodb mod file * update: fixed testcase for snos worker * chore: correct variable name * update: added support to check againt multiple status - is_worker_enabled, get_jobs_by_statuses * docs: rewrote 1 job per block assumption * docs: DataSubmissionWorker -> DataAvailabilitySynchronizer * chore: liniting fix * update: changed name : DataAvailabilitySynchronizer -> DataSubmissionWorker --------- Co-authored-by: Apoorv Sadana <95699312+apoorvsadana@users.noreply.github.com>

* feat: Aptos Settelement Layer (#7) * feat(aptos_settlement_client): update state kzg & update state & verify inclusion * test(aptos_settlement_client): add update_state, state_block_number unit test * fix: dependencies * refactor code * feat: add submodule ionia * feat(aptos_settlement_client): update state kzg & update state & verify inclusion * test(aptos_settlement_client): add update_state, state_block_number unit test * fix: dependencies * refactor code * feat: add submodule ionia * refactor code * refactor code --------- Co-authored-by: Steve Nguyen <sonntuet1997@gmail.com> * feat: pull request template * Bump ionia from `3a15977` to `fe4cd5a` (#10) Bumps [ionia](https://github.com/sota-zk-labs/ionia) from `3a15977` to `fe4cd5a`. - [Commits](sota-zk-labs/ionia@3a15977...fe4cd5a) --- updated-dependencies: - dependency-name: ionia dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feat: Aptos Da Client * feat: Data Submission Worker Integration. (madara-alliance#51) * update: DA job draft #1 * docs: changelog updated * update: is_worker_enabled impl & usage in da_submission, removal of String from VerificationFailed * update: renamed to * update: run worker only if it's enabled using is_worker_enabled check * build: linter fixes * Update CHANGELOG.md Co-authored-by: Apoorv Sadana <95699312+apoorvsadana@users.noreply.github.com> * update: limit_to_one on get_jobs_by_status * update: removed get_last_successful_job_by_type, added get_latest_job_by_type_and_status * update: added error to job metadata * update: pr resolution, simplifying get_jobs_by_status, rejected status in verify_jobs * update: linting fixes * Update crates/orchestrator/src/jobs/mod.rs Co-authored-by: Apoorv Sadana <95699312+apoorvsadana@users.noreply.github.com> * update: removing .expect from mongodb mod file * update: fixed testcase for snos worker * chore: correct variable name * update: added support to check againt multiple status - is_worker_enabled, get_jobs_by_statuses * docs: rewrote 1 job per block assumption * docs: DataSubmissionWorker -> DataAvailabilitySynchronizer * chore: liniting fix * update: changed name : DataAvailabilitySynchronizer -> DataSubmissionWorker --------- Co-authored-by: Apoorv Sadana <95699312+apoorvsadana@users.noreply.github.com> * Feat : add testing fixtures and a basic test (madara-alliance#52) * feat : added tests for increasing coverage * feat : added mongo db tests and fixtures and updated the ci for tests * update : removed unwanted fixtures * update : removed unwanted fixtures * update : added mongo db runner in ci * update : added mongo db runner in ci * update : added mongo db runner in ci * update : updated with new changes and ci * update : updated test cases for s3 client * update : added .env.test file in the commit * feat : added database necessary tests * feat : added database necessary tests * Revert "feat : added database necessary tests" This reverts commit 65d66e6. * Revert "feat : added database necessary tests" This reverts commit 906a1eb. * update: Replaced Build_Config Fixture with TestConfigBuilder * update : config update * update : test_put_and_get_data_s3 test * update: moved testconfigbuilder to tests/config.rs & added docs , drop all collections not just jobs * feat : updated test config and added config type to aws s3 config * chore: resolved pr comments * Update crates/orchestrator/src/tests/database/mod.rs Co-authored-by: 0xevolve <Artevolve@yahoo.com> * feat : lint fix * fix : coverage tests fix * fix : test fix --------- Co-authored-by: Heemank Verma <heemankv@gmail.com> Co-authored-by: 0xevolve <Artevolve@yahoo.com> * Feat : job handler tests (madara-alliance#67) * feat : added tests for increasing coverage * feat : added mongo db tests and fixtures and updated the ci for tests * update : removed unwanted fixtures * update : removed unwanted fixtures * update : added mongo db runner in ci * update : added mongo db runner in ci * update : added mongo db runner in ci * update : updated with new changes and ci * update : updated test cases for s3 client * update : added .env.test file in the commit * feat : added database necessary tests * feat : added database necessary tests * Revert "feat : added database necessary tests" This reverts commit 65d66e6. * Revert "feat : added database necessary tests" This reverts commit 906a1eb. * update: Replaced Build_Config Fixture with TestConfigBuilder * update : config update * update : test_put_and_get_data_s3 test * update: moved testconfigbuilder to tests/config.rs & added docs , drop all collections not just jobs * feat : create job test case error returning * mock job handler * feat : added jobs test and modified worker tests * feat : added queue checks to tests and revamped some tests * feat : updated tests and resolved comments * feat : updated test config and added config type to aws s3 config * feat : updated tests and test names * feat : lint fixes * feat : lint fixes * feat : lint fixes * chore: resolved pr comments * Update crates/orchestrator/src/tests/database/mod.rs Co-authored-by: 0xevolve <Artevolve@yahoo.com> * chore : lint fixes * feat : lint fix * fix : coverage tests fix * fix : test fix * fix : updated region in localstack .env.test * feat : updated region * debug : added debug log to github ci * feat : updated queue code for test fixes * fix : sqs region fix * debug : added debug logs for ci debugging * feat : added override endpoint to queue url in producer and consumer * feat : added override endpoint to queue url in producer and consumer * fix : removed logs and refactored the code * chore : refactor code --------- Co-authored-by: Heemank Verma <heemankv@gmail.com> Co-authored-by: apoorvsadana <95699312+apoorvsadana@users.noreply.github.com> Co-authored-by: 0xevolve <Artevolve@yahoo.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Steve Nguyen <sonntuet1997@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Heemank Verma <heemankv@gmail.com> Co-authored-by: Apoorv Sadana <95699312+apoorvsadana@users.noreply.github.com> Co-authored-by: Arun Jangra <arunjangra1001@gmail.com> Co-authored-by: 0xevolve <Artevolve@yahoo.com>

* update: DA job draft #1 * docs: changelog updated * update: is_worker_enabled impl & usage in da_submission, removal of String from VerificationFailed * update: renamed to * update: run worker only if it's enabled using is_worker_enabled check * build: linter fixes * Update CHANGELOG.md Co-authored-by: Apoorv Sadana <95699312+apoorvsadana@users.noreply.github.com> * update: limit_to_one on get_jobs_by_status * update: removed get_last_successful_job_by_type, added get_latest_job_by_type_and_status * update: added error to job metadata * update: pr resolution, simplifying get_jobs_by_status, rejected status in verify_jobs * update: linting fixes * Update crates/orchestrator/src/jobs/mod.rs Co-authored-by: Apoorv Sadana <95699312+apoorvsadana@users.noreply.github.com> * update: removing .expect from mongodb mod file * update: fixed testcase for snos worker * chore: correct variable name * update: added support to check againt multiple status - is_worker_enabled, get_jobs_by_statuses * docs: rewrote 1 job per block assumption * docs: DataSubmissionWorker -> DataAvailabilitySynchronizer * chore: liniting fix * update: changed name : DataAvailabilitySynchronizer -> DataSubmissionWorker --------- Co-authored-by: Apoorv Sadana <95699312+apoorvsadana@users.noreply.github.com>

update: DA job draft #1

2310fcf

heemankv added the enhancement New feature or request label Jul 22, 2024

heemankv requested a review from apoorvsadana July 22, 2024 13:17

heemankv self-assigned this Jul 22, 2024

docs: changelog updated

df4fa86

apoorvsadana reviewed Jul 22, 2024

View reviewed changes

crates/orchestrator/src/workers/data_submission.rs Outdated Show resolved Hide resolved

crates/orchestrator/src/workers/data_submission.rs Outdated Show resolved Hide resolved

crates/orchestrator/src/workers/data_submission.rs Outdated Show resolved Hide resolved

ocdbytes reviewed Jul 22, 2024

View reviewed changes

crates/orchestrator/src/workers/data_submission.rs Outdated Show resolved Hide resolved

heemankv added 5 commits July 23, 2024 12:21

update: is_worker_enabled impl & usage in da_submission, removal of S…

d21abff

…tring from VerificationFailed

update: renamed to

f9f4732

update: run worker only if it's enabled using is_worker_enabled check

55e1bc8

build: linter fixes

9d7ab21

Merge branch 'main' into feat/da_worker

b209c18

heemankv commented Jul 23, 2024

View reviewed changes

crates/orchestrator/src/workers/snos.rs Show resolved Hide resolved

apoorvsadana reviewed Jul 23, 2024

View reviewed changes

heemankv and others added 2 commits July 23, 2024 17:17

Update CHANGELOG.md

fbd721d

Co-authored-by: Apoorv Sadana <95699312+apoorvsadana@users.noreply.github.com>

update: limit_to_one on get_jobs_by_status

e475cc4

heemankv force-pushed the feat/da_worker branch from ce09eeb to e475cc4 Compare July 23, 2024 12:31

heemankv added 2 commits July 23, 2024 18:44

update: removed get_last_successful_job_by_type, added get_latest_job…

c13a576

…_by_type_and_status

update: added error to job metadata

e611040

heemankv force-pushed the feat/da_worker branch from f1e1a76 to e611040 Compare July 23, 2024 14:39

apoorvsadana reviewed Jul 24, 2024

View reviewed changes

heemankv added 2 commits July 25, 2024 11:52

update: pr resolution, simplifying get_jobs_by_status, rejected statu…

586c709

…s in verify_jobs

update: linting fixes

b0bf981

apoorvsadana reviewed Jul 25, 2024

View reviewed changes

crates/orchestrator/src/database/mongodb/mod.rs Show resolved Hide resolved

crates/orchestrator/src/database/mongodb/mod.rs Show resolved Hide resolved

crates/orchestrator/src/jobs/mod.rs Outdated Show resolved Hide resolved

heemankv and others added 3 commits July 25, 2024 19:15

Update crates/orchestrator/src/jobs/mod.rs

69b05de

Co-authored-by: Apoorv Sadana <95699312+apoorvsadana@users.noreply.github.com>

update: removing .expect from mongodb mod file

46b8c23

update: fixed testcase for snos worker

8cea54d

akhercha reviewed Jul 25, 2024

View reviewed changes

crates/orchestrator/src/workers/data_submission.rs Outdated Show resolved Hide resolved

akhercha reviewed Jul 25, 2024

View reviewed changes

crates/orchestrator/src/workers/mod.rs Outdated Show resolved Hide resolved

chore: correct variable name

579d1d1

EvolveArt reviewed Jul 25, 2024

View reviewed changes

heemankv added 5 commits July 25, 2024 23:15

update: added support to check againt multiple status - is_worker_ena…

992190a

…bled, get_jobs_by_statuses

docs: rewrote 1 job per block assumption

ee29e87

docs: DataSubmissionWorker -> DataAvailabilitySynchronizer

465241c

Merge branch 'main' into feat/da_worker

e4a2aa0

chore: liniting fix

4ecf4fc

apoorvsadana approved these changes Aug 2, 2024

View reviewed changes

update: changed name : DataAvailabilitySynchronizer -> DataSubmission…

5905f88

…Worker

apoorvsadana merged commit 43db46e into main Aug 3, 2024
7 checks passed

heemankv deleted the feat/da_worker branch August 3, 2024 10:42

heemankv mentioned this pull request Aug 8, 2024

Tests for DA job #61

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Data Submission Worker Integration. #51

feat: Data Submission Worker Integration. #51

heemankv commented Jul 22, 2024

apoorvsadana left a comment

feat: Data Submission Worker Integration. #51

feat: Data Submission Worker Integration. #51

Conversation

heemankv commented Jul 22, 2024

Includes creation of multiple data submission jobs together.

Logic :

apoorvsadana left a comment

Choose a reason for hiding this comment