Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

detect node being hard down after upgrade fails #19

Open
mkaczanowski opened this issue Nov 13, 2024 · 0 comments
Open

detect node being hard down after upgrade fails #19

mkaczanowski opened this issue Nov 13, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@mkaczanowski
Copy link
Contributor

TL;DR

With last sommelier upgrade, the node panicked at the migration step:

3:06PM INF cellarfees v1 to v2: Store migration complete module=server
3:06PM INF migrating module crisis from version 1 to version 2 module=server
3:06PM INF migrating module distribution from version 2 to version 3 module=server
3:06PM INF migrating module gov from version 3 to version 4 module=server  
panic: no concrete type registered for type URL /cork.v1.AddManagedCellarIDsProposal against interface *v1beta1.Content  

The node died, but the post check was at:

Post upgrade check: GRPC_RESPONSIVE Waiting for the grpc and cometbft services to be responsive

In the end the post check failed so we knew smth was off. But I would think it would be nice for GRPC response check to also check if the node is down down.

If the node is hard down (docker ps) and there is no restart policy (which there is none) we know the node won't come back online.

@mkaczanowski mkaczanowski added the enhancement New feature or request label Nov 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant