Code

blockstream store to disk#339

Code

Open

grooviegermanikus wants to merge 180 commits intomainblockworks-foundation/lite-rpc:mainfrom feature/blockstream_store_to_diskblockworks-foundation/lite-rpc:feature/blockstream_store_to_diskCopy head branch name to clipboard

+2,841-498

Commits

Commits on Jan 22, 2024

• put block sequence invariants in a debugging task

  grooviegermanikus

  committed
  3c34866

  Copy full SHA for 3c34866

• add to main

  grooviegermanikus

  committed
  b919309

  Copy full SHA for b919309

• switch to blockhash instead of slot

  grooviegermanikus

  committed
  f16adc8

  Copy full SHA for f16adc8

• update dependencies

  grooviegermanikus

  committed
  3960491

  Copy full SHA for 3960491

• WIP use autoconnect_task

  grooviegermanikus

  committed
  1dd9ef3

  Copy full SHA for 1dd9ef3

• merge into stream

  grooviegermanikus

  committed
  b372c6d

  Copy full SHA for b372c6d

• handle shutdown

  grooviegermanikus

  committed
  3df4972

  Copy full SHA for 3df4972

• cleanup

  grooviegermanikus

  committed
  a899c0d

  Copy full SHA for a899c0d

• use autoconnect also for blockmeta

  grooviegermanikus

  committed
  009d273

  Copy full SHA for 009d273

• use new autoconnect

  grooviegermanikus

  committed
  4055183

  Copy full SHA for 4055183

• deploy experimental

  grooviegermanikus

  committed
  b372bc4

  Copy full SHA for b372bc4

• enable experimental

  grooviegermanikus

  committed
  c56ec56

  Copy full SHA for c56ec56

• reverrt fly log

  grooviegermanikus

  committed
  c28b54c

  Copy full SHA for c28b54c

Commits on Jan 23, 2024

• wire up slot

  grooviegermanikus

  committed
  2105ed7

  Copy full SHA for 2105ed7

• small renamings

  grooviegermanikus

  committed
  fbcd730

  Copy full SHA for fbcd730

• use autoconnect for blocks too

  grooviegermanikus

  committed
  0ddd4db

  Copy full SHA for 0ddd4db

• remove staging region

  grooviegermanikus

  committed
  c7e7b67

  Copy full SHA for c7e7b67

• cleanup task

  grooviegermanikus

  committed
  da0fdee

  Copy full SHA for da0fdee

• FIX panic in multiplex

  grooviegermanikus

  committed
  4f6952c

  Copy full SHA for 4f6952c

• improve log

  grooviegermanikus

  committed
  fa9ca19

  Copy full SHA for fa9ca19

• handle timeout in slot stream

  grooviegermanikus

  committed
  0fc9b5b

  Copy full SHA for 0fc9b5b

Commits on Jan 24, 2024

• comment on task termination

  grooviegermanikus

  committed
  d44379b

  Copy full SHA for d44379b

• cleanup

  grooviegermanikus

  committed
  e681727

  Copy full SHA for e681727

• handle poll_vote_accounts_and_cluster_info errors

  grooviegermanikus

  committed
  a64e699

  Copy full SHA for a64e699

• handle poll_vote_accounts_and_cluster_info errors

  grooviegermanikus

  committed
  62f89f4

  Copy full SHA for 62f89f4

• remove debug code

  grooviegermanikus

  committed
  15f3dfa

  Copy full SHA for 15f3dfa

• add logic to store finalized meta and merge with full processed block

  grooviegermanikus

  committed
  ba40c70

  Copy full SHA for ba40c70

• add debug stream tasks

  grooviegermanikus

  committed
  ae17f20

  Copy full SHA for ae17f20

• check previous_blockhash

  grooviegermanikus

  committed
  cdeb2e2

  Copy full SHA for cdeb2e2

• improve shutdown

  grooviegermanikus

  committed
  0d1e81c

  Copy full SHA for 0d1e81c

• log liveness

  grooviegermanikus

  committed
  e9f2280

  Copy full SHA for e9f2280

Commits on Jan 25, 2024

• stream BlockMeta

  grooviegermanikus

  committed
  b4512e6

  Copy full SHA for b4512e6

Commits on Jan 29, 2024

• add error log

  grooviegermanikus

  committed
  9aed98a

  Copy full SHA for 9aed98a

Commits on Jan 31, 2024

• Merge remote-tracking branch 'origin/main' into improvement/check-block-sequencing

  grooviegermanikus

  committed
  8874da7

  Copy full SHA for 8874da7

• remove failing check

  grooviegermanikus

  committed
  ff30ecc

  Copy full SHA for ff30ecc

• clippy+fmt

  grooviegermanikus

  committed
  d494163

  Copy full SHA for d494163

Commits on Feb 6, 2024

• merge from main

  grooviegermanikus

  committed
  c1fbd0b

  Copy full SHA for c1fbd0b

• reduce log level to "warn"

  grooviegermanikus

  committed
  b76a6ef

  Copy full SHA for b76a6ef

• fmt

  grooviegermanikus

  committed
  63b682e

  Copy full SHA for 63b682e

• use tagged geyser-grpc-connector

  grooviegermanikus

  committed
  40eee04

  Copy full SHA for 40eee04

• fix cargo fmt

  grooviegermanikus

  committed
  7f210b4

  Copy full SHA for 7f210b4

Commits on Feb 7, 2024

• remove obsolete cleanup_without_recv_blocks

  grooviegermanikus

  committed
  9bad056

  Copy full SHA for 9bad056

• fix clippy

  grooviegermanikus

  committed
  d3df98e

  Copy full SHA for d3df98e

• fmt glitch

  grooviegermanikus

  committed
  175918e

  Copy full SHA for 175918e

• listen to blocks, serialize

  grooviegermanikus

  committed
  f8c622d

  Copy full SHA for f8c622d

• roundtrip

  grooviegermanikus

  committed
  6ffb7b0

  Copy full SHA for 6ffb7b0

Commits on Feb 9, 2024

• wip

  grooviegermanikus

  committed
  f3e0ea3

  Copy full SHA for f3e0ea3

Commits on Feb 26, 2024

• use dump helper

  grooviegermanikus

  committed
  9dc675f

  Copy full SHA for 9dc675f

• move grpc dumper to literpc

  grooviegermanikus

  committed
  09efd94

  Copy full SHA for 09efd94

• add decription for args

  grooviegermanikus

  committed
  17d847d

  Copy full SHA for 17d847d

• use write_all

  grooviegermanikus

  committed
  372c67c

  Copy full SHA for 372c67c

• fmt

  grooviegermanikus

  committed
  96d7dc2

  Copy full SHA for 96d7dc2

• clippy

  grooviegermanikus

  committed
  1bb3546

  Copy full SHA for 1bb3546

• claify error

  grooviegermanikus

  committed
  8908d7b

  Copy full SHA for 8908d7b

• Merge remote-tracking branch 'origin/main' into feature/blockstream_store_to_disk

  grooviegermanikus

  committed
  05a9726

  Copy full SHA for 05a9726

• pass commitment_config

  grooviegermanikus

  committed
  8139f3d

  Copy full SHA for 8139f3d

• ignore local test

  grooviegermanikus

  committed
  857dea8

  Copy full SHA for 857dea8

Commits on Mar 4, 2024

• Merge remote-tracking branch 'origin/main' into feature/blockstream_store_to_disk

  grooviegermanikus

  committed
  da14a8f

  Copy full SHA for da14a8f

• move bin to exapmles

  grooviegermanikus

  committed
  f36b92d

  Copy full SHA for f36b92d

• move grpc block dumper to examples crate

  grooviegermanikus

  committed
  66654c6

  Copy full SHA for 66654c6

• clippy

  grooviegermanikus

  committed
  fb65ae4

  Copy full SHA for fb65ae4

• cleanup lib from cargo.toml

  grooviegermanikus

  committed
  fdf93b4

  Copy full SHA for fdf93b4

• fix STOARGE issue with pre pg16

  grooviegermanikus

  committed
  1519c5f

  Copy full SHA for 1519c5f

• fix STORAGE DDL

  grooviegermanikus

  committed
  a059b3b

  Copy full SHA for a059b3b

• optimize block column ordering

  grooviegermanikus

  committed
  2b27887

  Copy full SHA for 2b27887

• reorder columns

  grooviegermanikus

  committed
  3af5531

  Copy full SHA for 3af5531

• add autovacuum settings

  grooviegermanikus

  committed
  d8300da

  Copy full SHA for d8300da

• fix typo

  grooviegermanikus

  committed
  6eac0e1

  Copy full SHA for 6eac0e1

• wire blockstore

  grooviegermanikus

  committed
  b71e7d5

  Copy full SHA for b71e7d5

• WIP: expose postgres on get_blocks RPC

  grooviegermanikus

  committed
  4ab483e

  Copy full SHA for 4ab483e

• very basic get_blocks logic

  grooviegermanikus

  committed
  dbee46d

  Copy full SHA for dbee46d

• add postgres block importer to main

  grooviegermanikus

  committed
  0283a6a

  Copy full SHA for 0283a6a

• clippy

  grooviegermanikus

  committed
  9a6ba06

  Copy full SHA for 9a6ba06

• db config

  grooviegermanikus

  committed
  06e03ae

  Copy full SHA for 06e03ae

• fix ddl

  grooviegermanikus

  committed
  c6581fd

  Copy full SHA for c6581fd

• reset connection (discard)

  grooviegermanikus

  committed
  1ce9a47

  Copy full SHA for 1ce9a47

• pg connection

  grooviegermanikus

  committed
  f104fce

  Copy full SHA for f104fce

• postgres version check

  grooviegermanikus

  committed
  94b3b63

  Copy full SHA for 94b3b63

Commits on Mar 5, 2024

• demo: life without sessioncache

  grooviegermanikus

  committed
  b212a7f

  Copy full SHA for b212a7f

• remove clone

  grooviegermanikus

  committed
  e4c5b8f

  Copy full SHA for e4c5b8f

• working version without pool

  grooviegermanikus

  committed
  32d02e6

  Copy full SHA for 32d02e6

• use dedicated create_transaction_ids method

  grooviegermanikus

  committed
  2f212b1

  Copy full SHA for 2f212b1

• varchar(88)

  grooviegermanikus

  committed
  404aea1

  Copy full SHA for 404aea1

• move recent_blockhash

  grooviegermanikus

  committed
  0c80c55

  Copy full SHA for 0c80c55

• compress text using lz4

  grooviegermanikus

  committed
  00049b9

  Copy full SHA for 00049b9

• fix version check

  grooviegermanikus

  committed
  0da6df2

  Copy full SHA for 0da6df2

• fix create table error

  grooviegermanikus

  committed
  f7c6a80

  Copy full SHA for f7c6a80

• use varchar(44) for blockhash

  Show description for 816701d
  grooviegermanikus

  committed
  816701d

  Copy full SHA for 816701d

Commits on Mar 6, 2024

• comment on join

  grooviegermanikus

  committed
  9beab6a

  Copy full SHA for 9beab6a

• make pg_config env more flexible

  grooviegermanikus

  committed
  38970bc

  Copy full SHA for 38970bc

• fix chunk logging

  grooviegermanikus

  committed
  2fe0b41

  Copy full SHA for 2fe0b41

• increase timeout after lot of reconnects on hetzner

  grooviegermanikus

  committed
  6c6a04c

  Copy full SHA for 6c6a04c

• remove iteration timer

  grooviegermanikus

  committed
  a4dffa2

  Copy full SHA for a4dffa2

• handle block stream closed

  grooviegermanikus

  committed
  b5d03a3

  Copy full SHA for b5d03a3

• compare testnet and local

  grooviegermanikus

  committed
  f9ca0f8

  Copy full SHA for f9ca0f8

• add rpc custom error mapping

  grooviegermanikus

  committed
  21d80a0

  Copy full SHA for 21d80a0

• flip mapper args

  grooviegermanikus

  committed
  a56f098

  Copy full SHA for a56f098

• map more errors

  grooviegermanikus

  committed
  45da593

  Copy full SHA for 45da593

• complete rpc error mapping

  grooviegermanikus

  committed
  9e27816

  Copy full SHA for 9e27816

• add index for transaction in block

  grooviegermanikus

  committed
  357d150

  Copy full SHA for 357d150

Commits on Mar 7, 2024

• bridge test

  grooviegermanikus

  committed
  32ae398

  Copy full SHA for 32ae398

• test with recent slot

  grooviegermanikus

  committed
  3582bac

  Copy full SHA for 3582bac

• cleanup

  grooviegermanikus

  committed
  881b2ff

  Copy full SHA for 881b2ff

• truncate transactions

  grooviegermanikus

  committed
  c8add5d

  Copy full SHA for c8add5d

• mapper test

  grooviegermanikus

  committed
  dda7958

  Copy full SHA for dda7958

• mapped UiConfirmedBlock

  grooviegermanikus

  committed
  9634150

  Copy full SHA for 9634150

• investigate transaction versioning

  grooviegermanikus

  committed
  07c80d8

  Copy full SHA for 07c80d8

• add comments on mapping todos

  grooviegermanikus

  committed
  8dba74b

  Copy full SHA for 8dba74b

• mapper test

  grooviegermanikus

  committed
  52862dd

  Copy full SHA for 52862dd

• wip tests

  grooviegermanikus

  committed
  bfe1abe

  Copy full SHA for bfe1abe

Commits on Mar 8, 2024

• green test versioned

  grooviegermanikus

  committed
  a25132c

  Copy full SHA for a25132c

• map legacy + v0

  grooviegermanikus

  committed
  c88bf9c

  Copy full SHA for c88bf9c

• track message_version

  grooviegermanikus

  committed
  b95fdc9

  Copy full SHA for b95fdc9

• soft fail on sql errors

  grooviegermanikus

  committed
  a2c9cef

  Copy full SHA for a2c9cef

• brush up migrations

  grooviegermanikus

  committed
  693f344

  Copy full SHA for 693f344

• map fee, pre_balances, post_balances

  grooviegermanikus

  committed
  a5ef43b

  Copy full SHA for a5ef43b

• map serializied inner instructions

  grooviegermanikus

  committed
  1fbaf65

  Copy full SHA for 1fbaf65

• fix create idx

  grooviegermanikus

  committed
  9a150ca

  Copy full SHA for 9a150ca

• map inner_instructions

  grooviegermanikus

  committed
  da46fd2

  Copy full SHA for da46fd2

• remove extra let

  grooviegermanikus

  committed
  7969cc1

  Copy full SHA for 7969cc1

• map log_messages, writable_accounts, readable_accounts

  grooviegermanikus

  committed
  7875709

  Copy full SHA for 7875709

Commits on Mar 11, 2024

• comment on mapping improvements

  grooviegermanikus

  committed
  3b07979

  Copy full SHA for 3b07979

• move integration test

  grooviegermanikus

  committed
  2b9c9d3

  Copy full SHA for 2b9c9d3

• fix mapping test

  grooviegermanikus

  committed
  06ce41f

  Copy full SHA for 06ce41f

• move mapping tests

  grooviegermanikus

  committed
  acc9724

  Copy full SHA for acc9724

• move mapping tests

  grooviegermanikus

  committed
  0041940

  Copy full SHA for 0041940

• cleanup tests

  grooviegermanikus

  committed
  f6c0e02

  Copy full SHA for f6c0e02

• cleanup tests

  grooviegermanikus

  committed
  5709cbc

  Copy full SHA for 5709cbc

• fix inner_instructions mapping

  grooviegermanikus

  committed
  7b5fdd9

  Copy full SHA for 7b5fdd9

• cleanup tests

  grooviegermanikus

  committed
  3a24ae6

  Copy full SHA for 3a24ae6

• serialize to jsonb/jsonb[]

  grooviegermanikus

  committed
  eff3bba

  Copy full SHA for eff3bba

• map pre_token_balances+post_token_balances

  grooviegermanikus

  committed
  34b9eb4

  Copy full SHA for 34b9eb4

• split query dto

  grooviegermanikus

  committed
  4a37c77

  Copy full SHA for 4a37c77

• Revert "split query dto"

  Show description for fe4bf63
  grooviegermanikus

  committed
  fe4bf63

  Copy full SHA for fe4bf63

Commits on Mar 12, 2024

• add prometheus literpc_blockstore_postgres to postgres_session

  grooviegermanikus

  committed
  caebac1

  Copy full SHA for caebac1

• format (+ add set statement_timeout)

  grooviegermanikus

  committed
  0ba567c

  Copy full SHA for 0ba567c

• workaround to map signatures without require full tx_with_meta

  grooviegermanikus

  committed
  524bd95

  Copy full SHA for 524bd95

Commits on Mar 13, 2024

• debug stmt for slow execute()

  grooviegermanikus

  committed
  5bc10fd

  Copy full SHA for 5bc10fd

• track timings without buckets

  grooviegermanikus

  committed
  52e56f3

  Copy full SHA for 52e56f3

• weak signature compression

  grooviegermanikus

  committed
  408ff5a

  Copy full SHA for 408ff5a

• increase autovacuum delays

  grooviegermanikus

  committed
  dc672aa

  Copy full SHA for dc672aa

• increase autovacuum delays

  grooviegermanikus

  committed
  db888d0

  Copy full SHA for db888d0

• set min_freeze_age=0

  grooviegermanikus

  committed
  2742251

  Copy full SHA for 2742251

• fix inner instruction mapping for rpc

  grooviegermanikus

  committed
  fdce38a

  Copy full SHA for fdce38a

• run optimize on every loop

  grooviegermanikus

  committed
  f8c6c34

  Copy full SHA for f8c6c34

• rename

  grooviegermanikus

  committed
  970caef

  Copy full SHA for 970caef

• optimizer impl

  grooviegermanikus

  committed
  6d66f78

  Copy full SHA for 6d66f78

• lock table in insert

  grooviegermanikus

  committed
  b85f586

  Copy full SHA for b85f586

• unique index fillfactor=80

  grooviegermanikus

  committed
  5bb47eb

  Copy full SHA for 5bb47eb

• use crazy create table returning pattern

  grooviegermanikus

  committed
  a9f6f80

  Copy full SHA for a9f6f80

• set lock_timeout

  grooviegermanikus

  committed
  e2a945d

  Copy full SHA for e2a945d

• use mapping table

  grooviegermanikus

  committed
  69f2947

  Copy full SHA for 69f2947

• add tracing

  grooviegermanikus

  committed
  60d0bfb

  Copy full SHA for 60d0bfb

• fmt

  grooviegermanikus

  committed
  37fbc34

  Copy full SHA for 37fbc34

• increase work_mem:

  Show description for ece0ad4
  grooviegermanikus

  committed
  ece0ad4

  Copy full SHA for ece0ad4

• ONLY 1 write_session

  grooviegermanikus

  committed
  e30c57a

  Copy full SHA for e30c57a

Commits on Mar 14, 2024

• format elapsed times as 0.00ms

  grooviegermanikus

  committed
  e112402

  Copy full SHA for e112402

• refactor chunk logic

  grooviegermanikus

  committed
  2268cd0

  Copy full SHA for 2268cd0

• print explain plan

  grooviegermanikus

  committed
  5018824

  Copy full SHA for 5018824

• add regex

  grooviegermanikus

  committed
  dac6d7b

  Copy full SHA for dac6d7b

• MODEL CHANGE: change pkey to (slot,transaction_id)

  grooviegermanikus

  committed
  c43d0f0

  Copy full SHA for c43d0f0

• improve write_speed test

  grooviegermanikus

  committed
  e8c2f04

  Copy full SHA for e8c2f04

• fix test

  grooviegermanikus

  committed
  49bd29f

  Copy full SHA for 49bd29f

• fix logging of timings

  grooviegermanikus

  committed
  2db819a

  Copy full SHA for 2db819a

• comment out execute_explain (citus does not like it)

  grooviegermanikus

  committed
  43ca986

  Copy full SHA for 43ca986

• add metric literpc_blockstore_save_block

  grooviegermanikus

  committed
  6666dda

  Copy full SHA for 6666dda

• remove confusing 'remaining 4 queue elements' log which happend due to non-confirmed blocks being queued

  grooviegermanikus

  committed
  d137e12

  Copy full SHA for d137e12

• set PARALLEL_WRITE_SESSIONS to 4

  grooviegermanikus

  committed
  647734e

  Copy full SHA for 647734e

• detect CITUS extension

  grooviegermanikus

  committed
  dfed888

  Copy full SHA for dfed888

• PARALLEL_WRITE_SESSIONS from 4 to 1

  grooviegermanikus

  committed
  2932997

  Copy full SHA for 2932997

• distribute by transaction_id instead of slot

  grooviegermanikus

  committed
  48ab0d6

  Copy full SHA for 48ab0d6

• add semikolon to pl sql

  grooviegermanikus

  committed
  3f67375

  Copy full SHA for 3f67375

• distribute by blocks.slot + transaction_ids.transaction_id

  grooviegermanikus

  committed
  10bf2e1

  Copy full SHA for 10bf2e1

• fix sql

  grooviegermanikus

  committed
  979e3d4

  Copy full SHA for 979e3d4

• cannot distribute transaction_ids

  grooviegermanikus

  committed
  3f02f6c

  Copy full SHA for 3f02f6c

Commits on Mar 15, 2024

• remove RETURNING from insert (citus does not like it)

  grooviegermanikus

  committed
  a594a40

  Copy full SHA for a594a40

• comment on work_mem

  grooviegermanikus

  committed
  05ab3eb

  Copy full SHA for 05ab3eb

• fix insert block check

  grooviegermanikus

  committed
  590b61d

  Copy full SHA for 590b61d

• add sigature as join to transaction_id JOIN not supported by citus

  grooviegermanikus

  committed
  fbbb2d5

  Copy full SHA for fbbb2d5

• cache epoch schema lookups

  grooviegermanikus

  committed
  2cfd044

  Copy full SHA for 2cfd044