[Bug]: pebbledb in app.toml doesn't work

[robert-zaremba]

Is there an existing issue for this?

• I have searched the existing issues

What happened?

When setting in app.toml:

app-db-backend = 'pebbledb'

and starting the simdv2 we observe the following error:

2:05AM INF Finalized block block_app_hash=A5B330B7D6E0A4729E2DF4F81CE2F979EE4DFE9215892AAAF00DC11D501DF7B4 height=1 module=state num_txs_res=0 num_val_updates=0 syncing_to_height=1
2:05AM INF Committed state block_app_hash=B9C57001AD8946F63A25C5FC9378381167D95C225EE6190CA95E01AC5B14A4DD height=1 module=state
2:05AM ERR Error in proxyAppConn.FinalizeBlock err="block delivery failed: failed to run beginblocker for authz: failed to get immutable tree at version 1: version does not exist" module=state
2:05AM ERR CONSENSUS FAILURE!!! err="failed to apply block; error block delivery failed: failed to run beginblocker for authz: failed to get immutable tree at version 1: version does not exist" module=consensus stack="goroutine 44 [running]:\nruntime/debug.Stack()\n\truntime/debug/stack.go:26 +0x5e\ngithub.com/cometbft/cometbft/internal/consensus.(*State).receiveRoutine.func2()\n\tgithub.com/cometbft/cometbft@v1.0.0/internal/consensus/state.go:805 +0x46\npanic({0x3ac7960?, 0xc002b3a630?})\n\truntime/panic.go:785 +0x132\ngithub.com/cometbft/cometbft/internal/consensus.(*State).finalizeCommit(0xc002a28008, 0x2)\n\tgithub.com/cometbft/cometbft@v1.0.0/internal/consensus/state.go:1925 +0xd25\ngithub.com/cometbft/cometbft/internal/consensus.(*State).tryFinalizeCommit(0xc002a28008, 0x2)\n\tgithub.com/cometbft/cometbft@v1.0.0/internal/consensus/state.go:1825 +0x349\ngithub.com/cometbft/cometbft/internal/consensus.(*State).enterCommit.func1()\n\tgithub.com/cometbft/cometbft@v1.0.0/internal/consensus/state.go:1760 +0x9c\ngithub.com/cometbft/cometbft/internal/consensus.(*State).enterCommit(0xc002a28008, 0x2, 0x0)\n\tgithub.com/cometbft/cometbft@v1.0.0/internal/consensus/state.go:1798 +0xc9b\ngithub.com/cometbft/cometbft/internal/consensus.(*State).addVote(0xc002a28008, 0xc002356a90, {0x0, 0x0})\n\tgithub.com/cometbft/cometbft@v1.0.0/internal/consensus/state.go:2488 +0x1d65\ngithub.com/cometbft/cometbft/internal/consensus.(*State).tryAddVote(0xc002a28008, 0xc002356a90, {0x0?, 0x4d028e?})\n\tgithub.com/cometbft/cometbft@v1.0.0/internal/consensus/state.go:2239 +0x29\ngithub.com/cometbft/cometbft/internal/consensus.(*State).handleMsg(0xc002a28008, {{0x4ad5380, 0xc00111c0a0}, {0x0, 0x0}, {0x0, 0x0, 0x0}})\n\tgithub.com/cometbft/cometbft@v1.0.0/internal/consensus/state.go:933 +0x425\ngithub.com/cometbft/cometbft/internal/consensus.(*State).receiveRoutine(0xc002a28008, 0x0)\n\tgithub.com/cometbft/cometbft@v1.0.0/internal/consensus/state.go:860 +0x50a\ncreated by github.com/cometbft/cometbft/internal/consensus.(*State).OnStart in goroutine 182\n\tgithub.com/cometbft/cometbft@v1.0.0/internal/consensus/state.go:402 +0x107\n"

NOTE: setting db_backend = "pebbledb" in config.toml works without problems.

Cosmos SDK Version

main(v2)

How to reproduce?

Basic setup

$APP init moniker --chain-id $cid

Update app.toml by setting:  app-db-backend = 'pebbledb'

$APP genesis add-genesis-account $val  $coins
$APP genesis gentx $val_name $delegation --chain-id $cid
$APP genesis collect-gentxs
$APP genesis validate
$APP start

[robert-zaremba]

In comet this works, so my guess is that something is wrong in cosmos-db.
Is there any reason we use cosmos-db rather than comet-db?

[tac0turtle]

the issue here i believe was around comet changing pebble implementations and how it effects us

[robert-zaremba]

But cometbft-db is not used in v0.52.

[robert-zaremba]

I confirm that the fix works. Thanks @tac0turtle

[robert-zaremba]

Reopening. Our validators reported the problem when syncing using pebbledb. While initially, the error appeared at height 1 (when launching any chain, even simapp), now the error appears later on when syncing.
Here are 2 reported errors:

Error 1

9:43AM INF Committed state block_app_hash=14D16127DAA7B1E65863CDD641711F5D19D61E7FE95C699B015E679650341D36 height=2200 module=state
9:43AM INF Finalized block block_app_hash=07E360FF576898968AA120761FF7C076D51C4B0930FDC0DB19A3815CB54F0107 height=2201 module=state num_txs_res=0 num_val_updates=0 syncing_to_height=116510
9:43AM ERR client error during proxyAppConn.CommitSync err="rpc error: code = Internal desc = collections: not found: key 'no_key' of type github.com/cosmos/gogoproto/cometbft.types.v1.ConsensusParams" module=state
panic: Failed to process committed block (2201:2DDEBF614268FD926960955AFCB81A8EBDC92E6680AFB67D849129348C00488E): commit failed for application: rpc error: code = Internal desc = collections: not found: key 'no_key' of type github.com/cosmos/gogoproto/cometbft.types.v1.ConsensusParams
goroutine 250 [running]:
github.com/cometbft/cometbft/internal/blocksync.(*Reactor).poolRoutine(0xc00202c488, 0x0)
    github.com/cometbft/cometbft@v1.0.0/internal/blocksync/reactor.go:551 +0x1845
github.com/cometbft/cometbft/internal/blocksync.(*Reactor).OnStart.func1()
    github.com/cometbft/cometbft@v1.0.0/internal/blocksync/reactor.go:139 +0x4e
created by github.com/cometbft/cometbft/internal/blocksync.(*Reactor).OnStart in goroutine 37
    github.com/cometbft/cometbft@v1.0.0/internal/blocksync/reactor.go:137 +0x8b
Error: exit status 2

Error 2:

9:51AM INF Committed state block_app_hash=AAC45EAB4BC9507E060DD03D5D79A4AF4C77AF72A2171F5312E0026BE751998D height=6701 module=state
9:51AM INF Finalized block block_app_hash=13C799AFF543F045ED8920F8CDD7EBD59FD75752FE6F0338F709D474FBDCAD20 height=6702 module=state num_txs_res=0 num_val_updates=0 syncing_to_height=116599
9:51AM INF Committed state block_app_hash=8ECA691AE620374DD2F0CC27339CA762BFEE7F7229E35AAC39AF4C56A5D88353 height=6702 module=state
9:51AM ERR Error in proxyAppConn.FinalizeBlock err="block delivery failed: failed to run endblock for gov: collections: not found: key 'no_key' of type github.com/cosmos/gogoproto/cosmos.gov.v1.Params" module=state
panic: Failed to process committed block (6703:9D4BB05F87BD1F0B1C7B93A01596260007EA504D6BE01E0D921165306B23CA37): block delivery failed: failed to run endblock for gov: collections: not found: key 'no_key' of type github.com/cosmos/gogoproto/cosmos.gov.v1.Params
goroutine 217 [running]:
github.com/cometbft/cometbft/internal/blocksync.(*Reactor).poolRoutine(0xc0028c2488, 0x0)
    github.com/cometbft/cometbft@v1.0.0/internal/blocksync/reactor.go:551 +0x1845
github.com/cometbft/cometbft/internal/blocksync.(*Reactor).OnStart.func1()
    github.com/cometbft/cometbft@v1.0.0/internal/blocksync/reactor.go:139 +0x4e
created by github.com/cometbft/cometbft/internal/blocksync.(*Reactor).OnStart in goroutine 163
    github.com/cometbft/cometbft@v1.0.0/internal/blocksync/reactor.go:137 +0x8b
Error: exit status 2

How to replicate

Try to sync Native testnet. Use pebbledb in app.yaml. Instructions.

Note:

• everything works well if we set goleveldb in app.yaml
• in config.yaml we use pebbledb and it also works well