Skip to content
This repository has been archived by the owner on Sep 26, 2022. It is now read-only.

BTCPay teos Tor #269

Open
uhlhosting opened this issue Mar 1, 2021 · 15 comments
Open

BTCPay teos Tor #269

uhlhosting opened this issue Mar 1, 2021 · 15 comments

Comments

@uhlhosting
Copy link

Hi,

It seems that teos integration in BTCPay-dockerized latest stable version is not working either via ipv4:port tor:port nothing works. Any idea where to start and or what to track?
@sr-gi
Copy link
Member

sr-gi commented Mar 3, 2021 •
edited
Loading

I'm giving a look at it at the moment.

Can you provide a docker log for the teos container so I can get a rough idea of what may be going on?

Should be something on the lines of: docker logs generated_bitcoin_teos_1

@uhlhosting
Copy link
Author

Here it is, some error keeps repeating from time to time, and the startup its weird also:

root@btc:~/btcpayserver-docker# docker logs generated_bitcoin_teos_1
27/02/2021 16:01:33 [Daemon] Tower identity found. Loading keys
27/02/2021 16:01:33 [Daemon] ERROR: An error occurred: Cannot connect to bitcoind. Shutting down
27/02/2021 16:01:37 [Daemon] Tower identity found. Loading keys
27/02/2021 16:01:41 [Daemon] Tower identity found. Loading keys
27/02/2021 16:01:41 [Daemon] ERROR: An error occurred: Cannot connect to bitcoind. Shutting down
27/02/2021 16:01:48 [Daemon] Tower identity found. Loading keys
27/02/2021 16:01:48 [Daemon] tower_id = 026247def18440eb3a3e59be829b7c12dd2e86da6989b88644de306132adf72761
27/02/2021 16:01:49 [Daemon] Starting TEOS
27/02/2021 16:01:49 [Daemon] Fresh bootstrap
27/02/2021 16:01:49 [Daemon] Internal API initialized. Serving at localhost:50051
[2021-02-27 16:01:49 +0000] [30] [INFO] Starting gunicorn 20.0.4
[2021-02-27 16:01:49 +0000] [30] [INFO] Listening at: http://0.0.0.0:9814 (30)
[2021-02-27 16:01:49 +0000] [30] [INFO] Using worker: sync
[2021-02-27 16:01:49 +0000] [31] [INFO] Booting worker with pid: 31
27/02/2021 16:01:49 [RPC] Initialized. Serving at localhost:8814
27/02/2021 16:01:49 [API] Initialized. Serving at 0.0.0.0:9814
27/02/2021 16:20:40 [ChainMonitor] New block received via zmq  (block_hash=00000000000000000000564444c8b005b5bb6942e6023844842b81a10bc8bf4b)
27/02/2021 16:20:41 [Watcher] New block received  (block_hash=00000000000000000000564444c8b005b5bb6942e6023844842b81a10bc8bf4b, prev_block_hash=00000000000000000008e2b41bf3415514e4e1a5d741971f38e75443d88046ee)
27/02/2021 16:20:41 [Responder] New block received  (block_hash=00000000000000000000564444c8b005b5bb6942e6023844842b81a10bc8bf4b, prev_block_hash=00000000000000000008e2b41bf3415514e4e1a5d741971f38e75443d88046ee)
27/02/2021 16:35:17 [ChainMonitor] New block received via zmq  (block_hash=000000000000000000049f53a64e4f30e5c7ae180ee6ff5f91a65c01340dc79e)
27/02/2021 16:35:17 [Responder] New block received  (block_hash=000000000000000000049f53a64e4f30e5c7ae180ee6ff5f91a65c01340dc79e, prev_block_hash=00000000000000000000564444c8b005b5bb6942e6023844842b81a10bc8bf4b)
27/02/2021 16:35:17 [Watcher] New block received  (block_hash=000000000000000000049f53a64e4f30e5c7ae180ee6ff5f91a65c01340dc79e, prev_block_hash=00000000000000000000564444c8b005b5bb6942e6023844842b81a10bc8bf4b)
27/02/2021 16:45:23 [ChainMonitor] New block received via zmq  (block_hash=00000000000000000000450cba6f5a4fabb3cf5294c63592c145adc2238722cf)
27/02/2021 16:45:23 [Watcher] New block received  (block_hash=00000000000000000000450cba6f5a4fabb3cf5294c63592c145adc2238722cf, prev_block_hash=000000000000000000049f53a64e4f30e5c7ae180ee6ff5f91a65c01340dc79e)
27/02/2021 16:45:23 [Responder] New block received  (block_hash=00000000000000000000450cba6f5a4fabb3cf5294c63592c145adc2238722cf, prev_block_hash=000000000000000000049f53a64e4f30e5c7ae180ee6ff5f91a65c01340dc79e)
27/02/2021 16:50:40 [ChainMonitor] New block received via zmq  (block_hash=0000000000000000000a853826cbab7fd1d283b074449f863b98700bc8580625)
27/02/2021 16:50:40 [Responder] New block received  (block_hash=0000000000000000000a853826cbab7fd1d283b074449f863b98700bc8580625, prev_block_hash=00000000000000000000450cba6f5a4fabb3cf5294c63592c145adc2238722cf)
27/02/2021 16:50:40 [Watcher] New block received  (block_hash=0000000000000000000a853826cbab7fd1d283b074449f863b98700bc8580625, prev_block_hash=00000000000000000000450cba6f5a4fabb3cf5294c63592c145adc2238722cf)
27/02/2021 17:03:29 [Daemon] Tower identity found. Loading keys
27/02/2021 17:03:29 [Daemon] tower_id = 026247def18440eb3a3e59be829b7c12dd2e86da6989b88644de306132adf72761
27/02/2021 17:03:29 [Daemon] Starting TEOS
27/02/2021 17:03:29 [Daemon] Fresh bootstrap
27/02/2021 17:03:29 [Daemon] Internal API initialized. Serving at localhost:50051
27/02/2021 17:03:30 [RPC] Initialized. Serving at localhost:8814
[2021-02-27 17:03:30 +0000] [30] [INFO] Starting gunicorn 20.0.4
[2021-02-27 17:03:30 +0000] [30] [INFO] Listening at: http://0.0.0.0:9814 (30)
[2021-02-27 17:03:30 +0000] [30] [INFO] Using worker: sync
[2021-02-27 17:03:30 +0000] [36] [INFO] Booting worker with pid: 36
27/02/2021 17:03:30 [API] Initialized. Serving at 0.0.0.0:9814
27/02/2021 17:14:34 [ChainMonitor] New block received via zmq  (block_hash=000000000000000000031dd30757501a8cd7374bec9a6edb2824c17aff5b4340)
27/02/2021 17:14:34 [Watcher] New block received  (block_hash=000000000000000000031dd30757501a8cd7374bec9a6edb2824c17aff5b4340, prev_block_hash=0000000000000000000a853826cbab7fd1d283b074449f863b98700bc8580625)
27/02/2021 17:14:34 [Responder] New block received  (block_hash=000000000000000000031dd30757501a8cd7374bec9a6edb2824c17aff5b4340, prev_block_hash=0000000000000000000a853826cbab7fd1d283b074449f863b98700bc8580625)
27/02/2021 17:14:54 [ChainMonitor] New block received via zmq  (block_hash=00000000000000000009b4a455271f63b53624b6f590a86ac6565e80e1c7e9be)
27/02/2021 17:14:54 [Responder] New block received  (block_hash=00000000000000000009b4a455271f63b53624b6f590a86ac6565e80e1c7e9be, prev_block_hash=000000000000000000031dd30757501a8cd7374bec9a6edb2824c17aff5b4340)
27/02/2021 17:14:54 [Watcher] New block received  (block_hash=00000000000000000009b4a455271f63b53624b6f590a86ac6565e80e1c7e9be, prev_block_hash=000000000000000000031dd30757501a8cd7374bec9a6edb2824c17aff5b4340)
27/02/2021 17:24:15 [ChainMonitor] New block received via zmq  (block_hash=00000000000000000008f4f145b65396f92cea6bb500b0d922ec94c1730dad83)
27/02/2021 17:24:15 [Responder] New block received  (block_hash=00000000000000000008f4f145b65396f92cea6bb500b0d922ec94c1730dad83, prev_block_hash=00000000000000000009b4a455271f63b53624b6f590a86ac6565e80e1c7e9be)
27/02/2021 17:24:15 [Watcher] New block received  (block_hash=00000000000000000008f4f145b65396f92cea6bb500b0d922ec94c1730dad83, prev_block_hash=00000000000000000009b4a455271f63b53624b6f590a86ac6565e80e1c7e9be)
27/02/2021 17:26:44 [ChainMonitor] New block received via zmq  (block_hash=00000000000000000008a3c9f745769d9a58ba321dc32c0a5ca54ae652c9f6aa)
27/02/2021 17:26:44 [Responder] New block received  (block_hash=00000000000000000008a3c9f745769d9a58ba321dc32c0a5ca54ae652c9f6aa, prev_block_hash=00000000000000000008f4f145b65396f92cea6bb500b0d922ec94c1730dad83)
27/02/2021 17:26:44 [Watcher] New block received  (block_hash=00000000000000000008a3c9f745769d9a58ba321dc32c0a5ca54ae652c9f6aa, prev_block_hash=00000000000000000008f4f145b65396f92cea6bb500b0d922ec94c1730dad83)
27/02/2021 17:29:29 [ChainMonitor] New block received via zmq  (block_hash=00000000000000000005ee92a6f587ff582dccdf717c7f5a8db1086c0da3d5fa)
27/02/2021 17:29:29 [Watcher] New block received  (block_hash=00000000000000000005ee92a6f587ff582dccdf717c7f5a8db1086c0da3d5fa, prev_block_hash=00000000000000000008a3c9f745769d9a58ba321dc32c0a5ca54ae652c9f6aa)
27/02/2021 17:29:29 [Responder] New block received  (block_hash=00000000000000000005ee92a6f587ff582dccdf717c7f5a8db1086c0da3d5fa, prev_block_hash=00000000000000000008a3c9f745769d9a58ba321dc32c0a5ca54ae652c9f6aa)
27/02/2021 17:36:29 [ChainMonitor] New block received via zmq  (block_hash=00000000000000000001cf0b8d58903d6bba81c93a1def6596f6ca78d81447c1)
27/02/2021 17:36:29 [Watcher] New block received  (block_hash=00000000000000000001cf0b8d58903d6bba81c93a1def6596f6ca78d81447c1, prev_block_hash=00000000000000000005ee92a6f587ff582dccdf717c7f5a8db1086c0da3d5fa)
27/02/2021 17:36:29 [Responder] New block received  (block_hash=00000000000000000001cf0b8d58903d6bba81c93a1def6596f6ca78d81447c1, prev_block_hash=00000000000000000005ee92a6f587ff582dccdf717c7f5a8db1086c0da3d5fa)
27/02/2021 17:51:25 [ChainMonitor] New block received via zmq  (block_hash=000000000000000000072237d3be5dd3f46837ee86542802a0ebb21e6c6968f6)
27/02/2021 17:51:25 [Watcher] New block received  (block_hash=000000000000000000072237d3be5dd3f46837ee86542802a0ebb21e6c6968f6, prev_block_hash=00000000000000000001cf0b8d58903d6bba81c93a1def6596f6ca78d81447c1)
27/02/2021 17:51:25 [Responder] New block received  (block_hash=000000000000000000072237d3be5dd3f46837ee86542802a0ebb21e6c6968f6, prev_block_hash=00000000000000000001cf0b8d58903d6bba81c93a1def6596f6ca78d81447c1)
27/02/2021 17:52:55 [ChainMonitor] New block received via zmq  (block_hash=0000000000000000000a8b4b4ebd9e3ec670fc99e3b86c7951590745d8fc645b)
27/02/2021 17:52:55 [Responder] New block received  (block_hash=0000000000000000000a8b4b4ebd9e3ec670fc99e3b86c7951590745d8fc645b, prev_block_hash=000000000000000000072237d3be5dd3f46837ee86542802a0ebb21e6c6968f6)
27/02/2021 17:52:55 [Watcher] New block received  (block_hash=0000000000000000000a8b4b4ebd9e3ec670fc99e3b86c7951590745d8fc645b, prev_block_hash=000000000000000000072237d3be5dd3f46837ee86542802a0ebb21e6c6968f6)
27/02/2021 18:01:26 [ChainMonitor] New block received via zmq  (block_hash=00000000000000000003563a63eaa804414c182eef2127fe4aae9e61f5a2138d)
27/02/2021 18:01:26 [Watcher] New block received  (block_hash=00000000000000000003563a63eaa804414c182eef2127fe4aae9e61f5a2138d, prev_block_hash=0000000000000000000a8b4b4ebd9e3ec670fc99e3b86c7951590745d8fc645b)
27/02/2021 18:01:26 [Responder] New block received  (block_hash=00000000000000000003563a63eaa804414c182eef2127fe4aae9e61f5a2138d, prev_block_hash=0000000000000000000a8b4b4ebd9e3ec670fc99e3b86c7951590745d8fc645b)
27/02/2021 18:08:16 [ChainMonitor] New block received via zmq  (block_hash=000000000000000000090ef0d38c6a5a8b7bf9348dbcb3dc801a0289fcd9bd29)
27/02/2021 18:08:16 [Responder] New block received  (block_hash=000000000000000000090ef0d38c6a5a8b7bf9348dbcb3dc801a0289fcd9bd29, prev_block_hash=00000000000000000003563a63eaa804414c182eef2127fe4aae9e61f5a2138d)
27/02/2021 18:08:16 [Watcher] New block received  (block_hash=000000000000000000090ef0d38c6a5a8b7bf9348dbcb3dc801a0289fcd9bd29, prev_block_hash=00000000000000000003563a63eaa804414c182eef2127fe4aae9e61f5a2138d)
27/02/2021 18:22:01 [ChainMonitor] New block received via zmq  (block_hash=00000000000000000004c423a33586a0d8dbbd2d4f1122d16cfe11742181b688)
27/02/2021 18:22:01 [Responder] New block received  (block_hash=00000000000000000004c423a33586a0d8dbbd2d4f1122d16cfe11742181b688, prev_block_hash=000000000000000000090ef0d38c6a5a8b7bf9348dbcb3dc801a0289fcd9bd29)
27/02/2021 18:22:01 [Watcher] New block received  (block_hash=00000000000000000004c423a33586a0d8dbbd2d4f1122d16cfe11742181b688, prev_block_hash=000000000000000000090ef0d38c6a5a8b7bf9348dbcb3dc801a0289fcd9bd29)
27/02/2021 18:54:30 [ChainMonitor] New block received via zmq  (block_hash=0000000000000000000081562e321c0528643d92508555707f02e702b3846578)
27/02/2021 18:54:30 [Responder] New block received  (block_hash=0000000000000000000081562e321c0528643d92508555707f02e702b3846578, prev_block_hash=00000000000000000004c423a33586a0d8dbbd2d4f1122d16cfe11742181b688)
27/02/2021 18:54:30 [Watcher] New block received  (block_hash=0000000000000000000081562e321c0528643d92508555707f02e702b3846578, prev_block_hash=00000000000000000004c423a33586a0d8dbbd2d4f1122d16cfe11742181b688)
27/02/2021 18:55:14 [ChainMonitor] New block received via zmq  (block_hash=00000000000000000006644ad08cd575c280aab60e2a8e67cf7bba703565d5aa)
27/02/2021 18:55:14 [Watcher] New block received  (block_hash=00000000000000000006644ad08cd575c280aab60e2a8e67cf7bba703565d5aa, prev_block_hash=0000000000000000000081562e321c0528643d92508555707f02e702b3846578)
27/02/2021 18:55:14 [Responder] New block received  (block_hash=00000000000000000006644ad08cd575c280aab60e2a8e67cf7bba703565d5aa, prev_block_hash=0000000000000000000081562e321c0528643d92508555707f02e702b3846578)
27/02/2021 18:56:24 [ChainMonitor] New block received via zmq  (block_hash=0000000000000000000b691cd63d88edb465e43b477948d9fd82d1e927b15eb6)
27/02/2021 18:56:24 [Responder] New block received  (block_hash=0000000000000000000b691cd63d88edb465e43b477948d9fd82d1e927b15eb6, prev_block_hash=00000000000000000006644ad08cd575c280aab60e2a8e67cf7bba703565d5aa)
27/02/2021 18:56:24 [Watcher] New block received  (block_hash=0000000000000000000b691cd63d88edb465e43b477948d9fd82d1e927b15eb6, prev_block_hash=00000000000000000006644ad08cd575c280aab60e2a8e67cf7bba703565d5aa)
27/02/2021 19:04:59 [ChainMonitor] New block received via zmq  (block_hash=00000000000000000001cadb8fe9824fd0778a551445ca3c82a8edd52ee78a2b)
27/02/2021 19:05:00 [Watcher] New block received  (block_hash=00000000000000000001cadb8fe9824fd0778a551445ca3c82a8edd52ee78a2b, prev_block_hash=0000000000000000000b691cd63d88edb465e43b477948d9fd82d1e927b15eb6)
27/02/2021 19:05:00 [Responder] New block received  (block_hash=00000000000000000001cadb8fe9824fd0778a551445ca3c82a8edd52ee78a2b, prev_block_hash=0000000000000000000b691cd63d88edb465e43b477948d9fd82d1e927b15eb6)
27/02/2021 19:22:32 [ChainMonitor] New block received via zmq  (block_hash=000000000000000000074b9d187cca8c5b69dc636f3640d04423705a10715d70)
27/02/2021 19:22:32 [Watcher] New block received  (block_hash=000000000000000000074b9d187cca8c5b69dc636f3640d04423705a10715d70, prev_block_hash=00000000000000000001cadb8fe9824fd0778a551445ca3c82a8edd52ee78a2b)
27/02/2021 19:22:32 [Responder] New block received  (block_hash=000000000000000000074b9d187cca8c5b69dc636f3640d04423705a10715d70, prev_block_hash=00000000000000000001cadb8fe9824fd0778a551445ca3c82a8edd52ee78a2b)
27/02/2021 19:34:47 [ChainMonitor] New block received via zmq  (block_hash=00000000000000000007d34ecc4de69e3f6659535669733ea349f1872095a3eb)
27/02/2021 19:34:47 [Watcher] New block received  (block_hash=00000000000000000007d34ecc4de69e3f6659535669733ea349f1872095a3eb, prev_block_hash=000000000000000000074b9d187cca8c5b69dc636f3640d04423705a10715d70)
27/02/2021 19:34:47 [Responder] New block received  (block_hash=00000000000000000007d34ecc4de69e3f6659535669733ea349f1872095a3eb, prev_block_hash=000000000000000000074b9d187cca8c5b69dc636f3640d04423705a10715d70)
27/02/2021 19:36:01 [ChainMonitor] New block received via zmq  (block_hash=0000000000000000000c0f2eb49170779b9277cdfaaead94395711ef242ee255)
27/02/2021 19:36:01 [Responder] New block received  (block_hash=0000000000000000000c0f2eb49170779b9277cdfaaead94395711ef242ee255, prev_block_hash=00000000000000000007d34ecc4de69e3f6659535669733ea349f1872095a3eb)
27/02/2021 19:36:01 [Watcher] New block received  (block_hash=0000000000000000000c0f2eb49170779b9277cdfaaead94395711ef242ee255, prev_block_hash=00000000000000000007d34ecc4de69e3f6659535669733ea349f1872095a3eb)
27/02/2021 19:37:38 [ChainMonitor] New block received via zmq  (block_hash=00000000000000000008c3e0d4288ef0d8b2474abd322fd35b5bed69a5cd4158)
27/02/2021 19:37:38 [Watcher] New block received  (block_hash=00000000000000000008c3e0d4288ef0d8b2474abd322fd35b5bed69a5cd4158, prev_block_hash=0000000000000000000c0f2eb49170779b9277cdfaaead94395711ef242ee255)
27/02/2021 19:37:38 [Responder] New block received  (block_hash=00000000000000000008c3e0d4288ef0d8b2474abd322fd35b5bed69a5cd4158, prev_block_hash=0000000000000000000c0f2eb49170779b9277cdfaaead94395711ef242ee255)
27/02/2021 19:43:18 [ChainMonitor] New block received via zmq  (block_hash=0000000000000000000182d8acc52e4cdc4a1b3480850e9904575f3ff8fc5873)
27/02/2021 19:43:18 [Responder] New block received  (block_hash=0000000000000000000182d8acc52e4cdc4a1b3480850e9904575f3ff8fc5873, prev_block_hash=00000000000000000008c3e0d4288ef0d8b2474abd322fd35b5bed69a5cd4158)
27/02/2021 19:43:18 [Watcher] New block received  (block_hash=0000000000000000000182d8acc52e4cdc4a1b3480850e9904575f3ff8fc5873, prev_block_hash=00000000000000000008c3e0d4288ef0d8b2474abd322fd35b5bed69a5cd4158)
27/02/2021 19:45:05 [ChainMonitor] New block received via zmq  (block_hash=0000000000000000000957793c7f9bc44044ddaac907591e2ef1ee36d5e8c448)
27/02/2021 19:45:05 [Responder] New block received  (block_hash=0000000000000000000957793c7f9bc44044ddaac907591e2ef1ee36d5e8c448, prev_block_hash=0000000000000000000182d8acc52e4cdc4a1b3480850e9904575f3ff8fc5873)
27/02/2021 19:45:05 [Watcher] New block received  (block_hash=0000000000000000000957793c7f9bc44044ddaac907591e2ef1ee36d5e8c448, prev_block_hash=0000000000000000000182d8acc52e4cdc4a1b3480850e9904575f3ff8fc5873)
27/02/2021 19:51:37 [ChainMonitor] New block received via zmq  (block_hash=00000000000000000003034338d5e29648e175ad3c85723045ff0524de7a7542)
27/02/2021 19:51:37 [Responder] New block received  (block_hash=00000000000000000003034338d5e29648e175ad3c85723045ff0524de7a7542, prev_block_hash=0000000000000000000957793c7f9bc44044ddaac907591e2ef1ee36d5e8c448)
27/02/2021 19:51:37 [Watcher] New block received  (block_hash=00000000000000000003034338d5e29648e175ad3c85723045ff0524de7a7542, prev_block_hash=0000000000000000000957793c7f9bc44044ddaac907591e2ef1ee36d5e8c448)
27/02/2021 20:01:51 [ChainMonitor] New block received via zmq  (block_hash=00000000000000000000368cd2eaab5ab9598b51f01ebce83ea8419232a66e9d)
27/02/2021 20:01:51 [Watcher] New block received  (block_hash=00000000000000000000368cd2eaab5ab9598b51f01ebce83ea8419232a66e9d, prev_block_hash=00000000000000000003034338d5e29648e175ad3c85723045ff0524de7a7542)
27/02/2021 20:01:51 [Responder] New block received  (block_hash=00000000000000000000368cd2eaab5ab9598b51f01ebce83ea8419232a66e9d, prev_block_hash=00000000000000000003034338d5e29648e175ad3c85723045ff0524de7a7542)
27/02/2021 20:03:56 [ChainMonitor] New block received via zmq  (block_hash=0000000000000000000b6549205de4024e92b9bdc21f87017adf1a01dc81a896)
27/02/2021 20:03:56 [Watcher] New block received  (block_hash=0000000000000000000b6549205de4024e92b9bdc21f87017adf1a01dc81a896, prev_block_hash=00000000000000000000368cd2eaab5ab9598b51f01ebce83ea8419232a66e9d)
27/02/2021 20:03:56 [Responder] New block received  (block_hash=0000000000000000000b6549205de4024e92b9bdc21f87017adf1a01dc81a896, prev_block_hash=00000000000000000000368cd2eaab5ab9598b51f01ebce83ea8419232a66e9d)
27/02/2021 20:26:05 [ChainMonitor] New block received via zmq  (block_hash=0000000000000000000cae534e4d44a2278f575e56a5f967ef18ba916a16806d)
27/02/2021 20:26:06 [Watcher] New block received  (block_hash=0000000000000000000cae534e4d44a2278f575e56a5f967ef18ba916a16806d, prev_block_hash=0000000000000000000b6549205de4024e92b9bdc21f87017adf1a01dc81a896)
27/02/2021 20:26:06 [Responder] New block received  (block_hash=0000000000000000000cae534e4d44a2278f575e56a5f967ef18ba916a16806d, prev_block_hash=0000000000000000000b6549205de4024e92b9bdc21f87017adf1a01dc81a896)
27/02/2021 20:37:22 [ChainMonitor] New block received via zmq  (block_hash=00000000000000000006a2a8f976a5ba4decd4bf724290b9db64fce935c7960d)
27/02/2021 20:37:22 [Watcher] New block received  (block_hash=00000000000000000006a2a8f976a5ba4decd4bf724290b9db64fce935c7960d, prev_block_hash=0000000000000000000cae534e4d44a2278f575e56a5f967ef18ba916a16806d)
27/02/2021 20:37:22 [Responder] New block received  (block_hash=00000000000000000006a2a8f976a5ba4decd4bf724290b9db64fce935c7960d, prev_block_hash=0000000000000000000cae534e4d44a2278f575e56a5f967ef18ba916a16806d)
27/02/2021 21:06:33 [ChainMonitor] New block received via zmq  (block_hash=00000000000000000007d701ed789c6831cec5bdc8ae859b13ecb71eb185a29e)
27/02/2021 21:06:33 [Watcher] New block received  (block_hash=00000000000000000007d701ed789c6831cec5bdc8ae859b13ecb71eb185a29e, prev_block_hash=00000000000000000006a2a8f976a5ba4decd4bf724290b9db64fce935c7960d)
27/02/2021 21:06:33 [Responder] New block received  (block_hash=00000000000000000007d701ed789c6831cec5bdc8ae859b13ecb71eb185a29e, prev_block_hash=00000000000000000006a2a8f976a5ba4decd4bf724290b9db64fce935c7960d)
27/02/2021 21:29:21 [ChainMonitor] New block received via zmq  (block_hash=0000000000000000000532fccdb8329d0ae1ba59c9f43471e6e4a1efe69e9100)
27/02/2021 21:29:21 [Watcher] New block received  (block_hash=0000000000000000000532fccdb8329d0ae1ba59c9f43471e6e4a1efe69e9100, prev_block_hash=00000000000000000007d701ed789c6831cec5bdc8ae859b13ecb71eb185a29e)
27/02/2021 21:29:21 [Responder] New block received  (block_hash=0000000000000000000532fccdb8329d0ae1ba59c9f43471e6e4a1efe69e9100, prev_block_hash=00000000000000000007d701ed789c6831cec5bdc8ae859b13ecb71eb185a29e)
27/02/2021 21:42:05 [ChainMonitor] New block received via zmq  (block_hash=00000000000000000009a5fbe17ed811f49ef03ae883cc88f6baeb51852ab3ac)
27/02/2021 21:42:05 [Watcher] New block received  (block_hash=00000000000000000009a5fbe17ed811f49ef03ae883cc88f6baeb51852ab3ac, prev_block_hash=0000000000000000000532fccdb8329d0ae1ba59c9f43471e6e4a1efe69e9100)
27/02/2021 21:42:05 [Responder] New block received  (block_hash=00000000000000000009a5fbe17ed811f49ef03ae883cc88f6baeb51852ab3ac, prev_block_hash=0000000000000000000532fccdb8329d0ae1ba59c9f43471e6e4a1efe69e9100)
27/02/2021 22:01:05 [ChainMonitor] New block received via zmq  (block_hash=00000000000000000006cae670f70127ec3d6422aad0d61d9984178c0caa2e02)
27/02/2021 22:01:05 [Watcher] New block received  (block_hash=00000000000000000006cae670f70127ec3d6422aad0d61d9984178c0caa2e02, prev_block_hash=00000000000000000009a5fbe17ed811f49ef03ae883cc88f6baeb51852ab3ac)
27/02/2021 22:01:05 [Responder] New block received  (block_hash=00000000000000000006cae670f70127ec3d6422aad0d61d9984178c0caa2e02, prev_block_hash=00000000000000000009a5fbe17ed811f49ef03ae883cc88f6baeb51852ab3ac)
27/02/2021 22:20:13 [ChainMonitor] New block received via zmq  (block_hash=0000000000000000000201084b7f4b42f95c1e01923dda308f944ee9ce936661)
27/02/2021 22:20:13 [Watcher] New block received  (block_hash=0000000000000000000201084b7f4b42f95c1e01923dda308f944ee9ce936661, prev_block_hash=00000000000000000006cae670f70127ec3d6422aad0d61d9984178c0caa2e02)
27/02/2021 22:20:13 [Responder] New block received  (block_hash=0000000000000000000201084b7f4b42f95c1e01923dda308f944ee9ce936661, prev_block_hash=00000000000000000006cae670f70127ec3d6422aad0d61d9984178c0caa2e02)
27/02/2021 22:35:00 [ChainMonitor] New block received via zmq  (block_hash=0000000000000000000c6ccf9473f54126cf1c1bbc2086e123dc26d816276a35)
27/02/2021 22:35:00 [Watcher] New block received  (block_hash=0000000000000000000c6ccf9473f54126cf1c1bbc2086e123dc26d816276a35, prev_block_hash=0000000000000000000201084b7f4b42f95c1e01923dda308f944ee9ce936661)
27/02/2021 22:35:00 [Responder] New block received  (block_hash=0000000000000000000c6ccf9473f54126cf1c1bbc2086e123dc26d816276a35, prev_block_hash=0000000000000000000201084b7f4b42f95c1e01923dda308f944ee9ce936661)
27/02/2021 22:57:31 [ChainMonitor] New block received via zmq  (block_hash=0000000000000000000c315e04e62f4a940ae70757c22b6bdaf6680efb5da013)
27/02/2021 22:57:31 [Watcher] New block received  (block_hash=0000000000000000000c315e04e62f4a940ae70757c22b6bdaf6680efb5da013, prev_block_hash=0000000000000000000c6ccf9473f54126cf1c1bbc2086e123dc26d816276a35)
27/02/2021 22:57:31 [Responder] New block received  (block_hash=0000000000000000000c315e04e62f4a940ae70757c22b6bdaf6680efb5da013, prev_block_hash=0000000000000000000c6ccf9473f54126cf1c1bbc2086e123dc26d816276a35)
27/02/2021 23:03:06 [ChainMonitor] New block received via zmq  (block_hash=00000000000000000009d4c70612485bd3a8f4fb05cc04bb12a47084884c0381)
27/02/2021 23:03:06 [Watcher] New block received  (block_hash=00000000000000000009d4c70612485bd3a8f4fb05cc04bb12a47084884c0381, prev_block_hash=0000000000000000000c315e04e62f4a940ae70757c22b6bdaf6680efb5da013)
27/02/2021 23:03:06 [Responder] New block received  (block_hash=00000000000000000009d4c70612485bd3a8f4fb05cc04bb12a47084884c0381, prev_block_hash=0000000000000000000c315e04e62f4a940ae70757c22b6bdaf6680efb5da013)
27/02/2021 23:08:02 [ChainMonitor] New block received via zmq  (block_hash=00000000000000000002d56ca740d0346c0c9522bd2636d5e260a975280c2588)
27/02/2021 23:08:02 [Watcher] New block received  (block_hash=00000000000000000002d56ca740d0346c0c9522bd2636d5e260a975280c2588, prev_block_hash=00000000000000000009d4c70612485bd3a8f4fb05cc04bb12a47084884c0381)
27/02/2021 23:08:02 [Responder] New block received  (block_hash=00000000000000000002d56ca740d0346c0c9522bd2636d5e260a975280c2588, prev_block_hash=00000000000000000009d4c70612485bd3a8f4fb05cc04bb12a47084884c0381)
27/02/2021 23:11:10 [ChainMonitor] New block received via zmq  (block_hash=000000000000000000077b8a2453eb2544db153dcf9d17923b17e6d194a50934)
27/02/2021 23:11:10 [Watcher] New block received  (block_hash=000000000000000000077b8a2453eb2544db153dcf9d17923b17e6d194a50934, prev_block_hash=00000000000000000002d56ca740d0346c0c9522bd2636d5e260a975280c2588)
27/02/2021 23:11:10 [Responder] New block received  (block_hash=000000000000000000077b8a2453eb2544db153dcf9d17923b17e6d194a50934, prev_block_hash=00000000000000000002d56ca740d0346c0c9522bd2636d5e260a975280c2588)
27/02/2021 23:25:14 [ChainMonitor] New block received via zmq  (block_hash=0000000000000000000745d736aa6e6d548d17006aaa84a9204801e05735377f)
27/02/2021 23:25:14 [Responder] New block received  (block_hash=0000000000000000000745d736aa6e6d548d17006aaa84a9204801e05735377f, prev_block_hash=000000000000000000077b8a2453eb2544db153dcf9d17923b17e6d194a50934)
27/02/2021 23:25:14 [Watcher] New block received  (block_hash=0000000000000000000745d736aa6e6d548d17006aaa84a9204801e05735377f, prev_block_hash=000000000000000000077b8a2453eb2544db153dcf9d17923b17e6d194a50934)
27/02/2021 23:25:36 [ChainMonitor] New block received via zmq  (block_hash=0000000000000000000bddacd380e2d85146dbba5ae2954a31a9ff3e32ae1ad3)
27/02/2021 23:25:36 [Responder] New block received  (block_hash=0000000000000000000bddacd380e2d85146dbba5ae2954a31a9ff3e32ae1ad3, prev_block_hash=0000000000000000000745d736aa6e6d548d17006aaa84a9204801e05735377f)
27/02/2021 23:25:36 [Watcher] New block received  (block_hash=0000000000000000000bddacd380e2d85146dbba5ae2954a31a9ff3e32ae1ad3, prev_block_hash=0000000000000000000745d736aa6e6d548d17006aaa84a9204801e05735377f)
27/02/2021 23:54:08 [ChainMonitor] New block received via zmq  (block_hash=0000000000000000000c03f035c13f6de01236875ff96d121ac177396b3899f5)
27/02/2021 23:54:08 [Responder] New block received  (block_hash=0000000000000000000c03f035c13f6de01236875ff96d121ac177396b3899f5, prev_block_hash=0000000000000000000bddacd380e2d85146dbba5ae2954a31a9ff3e32ae1ad3)
27/02/2021 23:54:08 [Watcher] New block received  (block_hash=0000000000000000000c03f035c13f6de01236875ff96d121ac177396b3899f5, prev_block_hash=0000000000000000000bddacd380e2d85146dbba5ae2954a31a9ff3e32ae1ad3)
28/02/2021 00:14:58 [ChainMonitor] New block received via zmq  (block_hash=0000000000000000000766eaffd467cc9a5c54d740ddf3c11283bc76f5dcd279)
28/02/2021 00:14:58 [Responder] New block received  (block_hash=0000000000000000000766eaffd467cc9a5c54d740ddf3c11283bc76f5dcd279, prev_block_hash=0000000000000000000c03f035c13f6de01236875ff96d121ac177396b3899f5)
28/02/2021 00:14:58 [Watcher] New block received  (block_hash=0000000000000000000766eaffd467cc9a5c54d740ddf3c11283bc76f5dcd279, prev_block_hash=0000000000000000000c03f035c13f6de01236875ff96d121ac177396b3899f5)
28/02/2021 00:33:00 [ChainMonitor] New block received via zmq  (block_hash=00000000000000000007b619218d276894e8768876fd42f81131a98dc000da77)
28/02/2021 00:33:00 [Responder] New block received  (block_hash=00000000000000000007b619218d276894e8768876fd42f81131a98dc000da77, prev_block_hash=0000000000000000000766eaffd467cc9a5c54d740ddf3c11283bc76f5dcd279)
28/02/2021 00:33:00 [Watcher] New block received  (block_hash=00000000000000000007b619218d276894e8768876fd42f81131a98dc000da77, prev_block_hash=0000000000000000000766eaffd467cc9a5c54d740ddf3c11283bc76f5dcd279)
28/02/2021 00:55:22 [ChainMonitor] New block received via zmq  (block_hash=0000000000000000000c428a229a3c23d084cc800002faaff801a5c34edcd837)
28/02/2021 00:55:22 [Responder] New block received  (block_hash=0000000000000000000c428a229a3c23d084cc800002faaff801a5c34edcd837, prev_block_hash=00000000000000000007b619218d276894e8768876fd42f81131a98dc000da77)
28/02/2021 00:55:22 [Watcher] New block received  (block_hash=0000000000000000000c428a229a3c23d084cc800002faaff801a5c34edcd837, prev_block_hash=00000000000000000007b619218d276894e8768876fd42f81131a98dc000da77)
28/02/2021 01:03:04 [ChainMonitor] New block received via zmq  (block_hash=0000000000000000000cdd9dfa6b93e9b7e190e23c656b79013e644fb7bbbab3)
28/02/2021 01:03:04 [Responder] New block received  (block_hash=0000000000000000000cdd9dfa6b93e9b7e190e23c656b79013e644fb7bbbab3, prev_block_hash=0000000000000000000c428a229a3c23d084cc800002faaff801a5c34edcd837)
28/02/2021 01:03:04 [Watcher] New block received  (block_hash=0000000000000000000cdd9dfa6b93e9b7e190e23c656b79013e644fb7bbbab3, prev_block_hash=0000000000000000000c428a229a3c23d084cc800002faaff801a5c34edcd837)
28/02/2021 01:05:59 [ChainMonitor] New block received via zmq  (block_hash=0000000000000000000ce5008ba21432ae8bc5e3ef1ade5c7a9b810391012643)
28/02/2021 01:05:59 [Responder] New block received  (block_hash=0000000000000000000ce5008ba21432ae8bc5e3ef1ade5c7a9b810391012643, prev_block_hash=0000000000000000000cdd9dfa6b93e9b7e190e23c656b79013e644fb7bbbab3)
28/02/2021 01:05:59 [Watcher] New block received  (block_hash=0000000000000000000ce5008ba21432ae8bc5e3ef1ade5c7a9b810391012643, prev_block_hash=0000000000000000000cdd9dfa6b93e9b7e190e23c656b79013e644fb7bbbab3)
28/02/2021 01:08:40 [ChainMonitor] New block received via zmq  (block_hash=0000000000000000000670dd30be6229e99eb13e3b163fa30990f8c3f2a05617)
28/02/2021 01:08:40 [Watcher] New block received  (block_hash=0000000000000000000670dd30be6229e99eb13e3b163fa30990f8c3f2a05617, prev_block_hash=0000000000000000000ce5008ba21432ae8bc5e3ef1ade5c7a9b810391012643)
28/02/2021 01:08:40 [Responder] New block received  (block_hash=0000000000000000000670dd30be6229e99eb13e3b163fa30990f8c3f2a05617, prev_block_hash=0000000000000000000ce5008ba21432ae8bc5e3ef1ade5c7a9b810391012643)
28/02/2021 01:14:31 [ChainMonitor] New block received via zmq  (block_hash=0000000000000000000bdcbb5ffe27549ef9c4ee0320aeaea6aad918fbfa6d16)
28/02/2021 01:14:32 [Responder] New block received  (block_hash=0000000000000000000bdcbb5ffe27549ef9c4ee0320aeaea6aad918fbfa6d16, prev_block_hash=0000000000000000000670dd30be6229e99eb13e3b163fa30990f8c3f2a05617)
28/02/2021 01:14:32 [Watcher] New block received  (block_hash=0000000000000000000bdcbb5ffe27549ef9c4ee0320aeaea6aad918fbfa6d16, prev_block_hash=0000000000000000000670dd30be6229e99eb13e3b163fa30990f8c3f2a05617)
28/02/2021 01:37:28 [ChainMonitor] New block received via zmq  (block_hash=000000000000000000015b3cdd37fd72df65a5e846a35bb7e6a9c60b589bf22c)
28/02/2021 01:37:28 [Responder] New block received  (block_hash=000000000000000000015b3cdd37fd72df65a5e846a35bb7e6a9c60b589bf22c, prev_block_hash=0000000000000000000bdcbb5ffe27549ef9c4ee0320aeaea6aad918fbfa6d16)
28/02/2021 01:37:28 [Watcher] New block received  (block_hash=000000000000000000015b3cdd37fd72df65a5e846a35bb7e6a9c60b589bf22c, prev_block_hash=0000000000000000000bdcbb5ffe27549ef9c4ee0320aeaea6aad918fbfa6d16)
28/02/2021 01:39:59 [ChainMonitor] New block received via zmq  (block_hash=000000000000000000087fe6f18a447440cb554ad13bcab19ceaf182849b7cc3)
28/02/2021 01:40:00 [Watcher] New block received  (block_hash=000000000000000000087fe6f18a447440cb554ad13bcab19ceaf182849b7cc3, prev_block_hash=000000000000000000015b3cdd37fd72df65a5e846a35bb7e6a9c60b589bf22c)
28/02/2021 01:40:00 [Responder] New block received  (block_hash=000000000000000000087fe6f18a447440cb554ad13bcab19ceaf182849b7cc3, prev_block_hash=000000000000000000015b3cdd37fd72df65a5e846a35bb7e6a9c60b589bf22c)
28/02/2021 02:08:56 [ChainMonitor] New block received via zmq  (block_hash=00000000000000000009a997551d19d2d4aabe245f7ffabea5d2e8425e6aef78)
28/02/2021 02:08:56 [Watcher] New block received  (block_hash=00000000000000000009a997551d19d2d4aabe245f7ffabea5d2e8425e6aef78, prev_block_hash=000000000000000000087fe6f18a447440cb554ad13bcab19ceaf182849b7cc3)
28/02/2021 02:08:56 [Responder] New block received  (block_hash=00000000000000000009a997551d19d2d4aabe245f7ffabea5d2e8425e6aef78, prev_block_hash=000000000000000000087fe6f18a447440cb554ad13bcab19ceaf182849b7cc3)
28/02/2021 02:09:28 [ChainMonitor] New block received via zmq  (block_hash=00000000000000000000f5a0080335e00891b96c5db64313e8d116728437fa29)
28/02/2021 02:09:28 [Responder] New block received  (block_hash=00000000000000000000f5a0080335e00891b96c5db64313e8d116728437fa29, prev_block_hash=00000000000000000009a997551d19d2d4aabe245f7ffabea5d2e8425e6aef78)
28/02/2021 02:09:28 [Watcher] New block received  (block_hash=00000000000000000000f5a0080335e00891b96c5db64313e8d116728437fa29, prev_block_hash=00000000000000000009a997551d19d2d4aabe245f7ffabea5d2e8425e6aef78)
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/threading.py", line 950, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.9/threading.py", line 888, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.9/site-packages/teos/chain_monitor.py", line 123, in monitor_chain_polling
    current_tip = self.block_processor.get_best_block_hash()
  File "/usr/local/lib/python3.9/site-packages/teos/block_processor.py", line 62, in get_best_block_hash
    block_hash = bitcoin_cli(self.btc_connect_params).getbestblockhash()
  File "/usr/local/lib/python3.9/site-packages/teos/utils/auth_proxy.py", line 151, in __call__
    response, status = self._request("POST", self.__url.path, postdata.encode("utf-8"))
  File "/usr/local/lib/python3.9/site-packages/teos/utils/auth_proxy.py", line 114, in _request
    self.__conn.request(method, path, postdata, headers)
  File "/usr/local/lib/python3.9/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.9/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.9/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.9/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.9/http/client.py", line 950, in send
    self.connect()
  File "/usr/local/lib/python3.9/http/client.py", line 921, in connect
    self.sock = self._create_connection(
  File "/usr/local/lib/python3.9/socket.py", line 843, in create_connection
    raise err
  File "/usr/local/lib/python3.9/socket.py", line 831, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
28/02/2021 02:13:42 [Daemon] Tower identity found. Loading keys
28/02/2021 02:13:42 [Daemon] tower_id = 026247def18440eb3a3e59be829b7c12dd2e86da6989b88644de306132adf72761
28/02/2021 02:13:44 [Daemon] Starting TEOS
28/02/2021 02:13:45 [Daemon] Fresh bootstrap
28/02/2021 02:13:45 [Daemon] Internal API initialized. Serving at localhost:50051
28/02/2021 02:13:45 [RPC] Initialized. Serving at localhost:8814
[2021-02-28 02:13:46 +0000] [30] [INFO] Starting gunicorn 20.0.4
[2021-02-28 02:13:46 +0000] [30] [INFO] Listening at: http://0.0.0.0:9814 (30)
[2021-02-28 02:13:46 +0000] [30] [INFO] Using worker: sync
[2021-02-28 02:13:46 +0000] [36] [INFO] Booting worker with pid: 36
28/02/2021 02:13:46 [API] Initialized. Serving at 0.0.0.0:9814
28/02/2021 02:18:01 [ChainMonitor] New block received via zmq  (block_hash=0000000000000000000ba5232523408775fd70b9d731200267e6d36073b9186f)
28/02/2021 02:18:01 [Responder] New block received  (block_hash=0000000000000000000ba5232523408775fd70b9d731200267e6d36073b9186f, prev_block_hash=00000000000000000000f5a0080335e00891b96c5db64313e8d116728437fa29)
28/02/2021 02:18:01 [Watcher] New block received  (block_hash=0000000000000000000ba5232523408775fd70b9d731200267e6d36073b9186f, prev_block_hash=00000000000000000000f5a0080335e00891b96c5db64313e8d116728437fa29)
28/02/2021 02:19:10 [ChainMonitor] New block received via zmq  (block_hash=0000000000000000000434a0210f460be762de3a65b79517b9d537a8b10c4c71)
28/02/2021 02:19:10 [Responder] New block received  (block_hash=0000000000000000000434a0210f460be762de3a65b79517b9d537a8b10c4c71, prev_block_hash=0000000000000000000ba5232523408775fd70b9d731200267e6d36073b9186f)
28/02/2021 02:19:10 [Watcher] New block received  (block_hash=0000000000000000000434a0210f460be762de3a65b79517b9d537a8b10c4c71, prev_block_hash=0000000000000000000ba5232523408775fd70b9d731200267e6d36073b9186f)
28/02/2021 02:43:49 [ChainMonitor] New block received via zmq  (block_hash=0000000000000000000136c6998cad74e2252c6295a0969fc250168e1fe78650)
28/02/2021 02:43:49 [Watcher] New block received  (block_hash=0000000000000000000136c6998cad74e2252c6295a0969fc250168e1fe78650, prev_block_hash=0000000000000000000434a0210f460be762de3a65b79517b9d537a8b10c4c71)
28/02/2021 02:43:49 [Responder] New block received  (block_hash=0000000000000000000136c6998cad74e2252c6295a0969fc250168e1fe78650, prev_block_hash=0000000000000000000434a0210f460be762de3a65b79517b9d537a8b10c4c71)
28/02/2021 02:49:31 [ChainMonitor] New block received via zmq  (block_hash=00000000000000000001a6277192705cdf28da247c64bd942d261f944a923d85)
28/02/2021 02:49:31 [Watcher] New block received  (block_hash=00000000000000000001a6277192705cdf28da247c64bd942d261f944a923d85, prev_block_hash=0000000000000000000136c6998cad74e2252c6295a0969fc250168e1fe78650)
28/02/2021 02:49:31 [Responder] New block received  (block_hash=00000000000000000001a6277192705cdf28da247c64bd942d261f944a923d85, prev_block_hash=0000000000000000000136c6998cad74e2252c6295a0969fc250168e1fe78650)
28/02/2021 02:50:41 [ChainMonitor] New block received via zmq  (block_hash=0000000000000000000012dfc019be41ebffb805d43561327d667764e677c06e)
28/02/2021 02:50:41 [Watcher] New block received  (block_hash=0000000000000000000012dfc019be41ebffb805d43561327d667764e677c06e, prev_block_hash=00000000000000000001a6277192705cdf28da247c64bd942d261f944a923d85)
28/02/2021 02:50:41 [Responder] New block received  (block_hash=0000000000000000000012dfc019be41ebffb805d43561327d667764e677c06e, prev_block_hash=00000000000000000001a6277192705cdf28da247c64bd942d261f944a923d85)
28/02/2021 02:52:38 [ChainMonitor] New block received via zmq  (block_hash=00000000000000000005f1b6b247dfaa6f6856686ae527b4ed0781ba18024daf)
28/02/2021 02:52:38 [Watcher] New block received  (block_hash=00000000000000000005f1b6b247dfaa6f6856686ae527b4ed0781ba18024daf, prev_block_hash=0000000000000000000012dfc019be41ebffb805d43561327d667764e677c06e)
28/02/2021 02:52:38 [Responder] New block received  (block_hash=00000000000000000005f1b6b247dfaa6f6856686ae527b4ed0781ba18024daf, prev_block_hash=0000000000000000000012dfc019be41ebffb805d43561327d667764e677c06e)
28/02/2021 03:01:05 [ChainMonitor] New block received via zmq  (block_hash=000000000000000000004af116f2d8b8ae5d0e4e903bc66b5e69cd8ff58ff7b0)
28/02/2021 03:01:05 [Responder] New block received  (block_hash=000000000000000000004af116f2d8b8ae5d0e4e903bc66b5e69cd8ff58ff7b0, prev_block_hash=00000000000000000005f1b6b247dfaa6f6856686ae527b4ed0781ba18024daf)
28/02/2021 03:01:05 [Watcher] New block received  (block_hash=000000000000000000004af116f2d8b8ae5d0e4e903bc66b5e69cd8ff58ff7b0, prev_block_hash=00000000000000000005f1b6b247dfaa6f6856686ae527b4ed0781ba18024daf)
28/02/2021 03:19:30 [ChainMonitor] New block received via zmq  (block_hash=00000000000000000001c526e814d73a729d98ada0874ba1f2c8da8a6e3b5cf2)
28/02/2021 03:19:30 [Watcher] New block received  (block_hash=00000000000000000001c526e814d73a729d98ada0874ba1f2c8da8a6e3b5cf2, prev_block_hash=000000000000000000004af116f2d8b8ae5d0e4e903bc66b5e69cd8ff58ff7b0)
28/02/2021 03:19:30 [Responder] New block received  (block_hash=00000000000000000001c526e814d73a729d98ada0874ba1f2c8da8a6e3b5cf2, prev_block_hash=000000000000000000004af116f2d8b8ae5d0e4e903bc66b5e69cd8ff58ff7b0)
28/02/2021 03:20:01 [ChainMonitor] New block received via zmq  (block_hash=000000000000000000095ee2df60c299445cb226536a993a162781ecf5a70fcc)
28/02/2021 03:20:01 [Responder] New block received  (block_hash=000000000000000000095ee2df60c299445cb226536a993a162781ecf5a70fcc, prev_block_hash=00000000000000000001c526e814d73a729d98ada0874ba1f2c8da8a6e3b5cf2)
28/02/2021 03:20:01 [Watcher] New block received  (block_hash=000000000000000000095ee2df60c299445cb226536a993a162781ecf5a70fcc, prev_block_hash=00000000000000000001c526e814d73a729d98ada0874ba1f2c8da8a6e3b5cf2)
28/02/2021 03:21:08 [ChainMonitor] New block received via zmq  (block_hash=000000000000000000046547ec9b1e8fe05446add5b0e54cab4222e85b02f087)
28/02/2021 03:21:08 [Responder] New block received  (block_hash=000000000000000000046547ec9b1e8fe05446add5b0e54cab4222e85b02f087, prev_block_hash=000000000000000000095ee2df60c299445cb226536a993a162781ecf5a70fcc)
28/02/2021 03:21:08 [Watcher] New block received  (block_hash=000000000000000000046547ec9b1e8fe05446add5b0e54cab4222e85b02f087, prev_block_hash=000000000000000000095ee2df60c299445cb226536a993a162781ecf5a70fcc)
28/02/2021 03:40:32 [ChainMonitor] New block received via zmq  (block_hash=0000000000000000000abcd2c9949ffe2363d83f56a041cf31a8dd74af7d9882)
28/02/2021 03:40:32 [Responder] New block received  (block_hash=0000000000000000000abcd2c9949ffe2363d83f56a041cf31a8dd74af7d9882, prev_block_hash=000000000000000000046547ec9b1e8fe05446add5b0e54cab4222e85b02f087)
28/02/2021 03:40:32 [Watcher] New block received  (block_hash=0000000000000000000abcd2c9949ffe2363d83f56a041cf31a8dd74af7d9882, prev_block_hash=000000000000000000046547ec9b1e8fe05446add5b0e54cab4222e85b02f087)
28/02/2021 03:56:06 [ChainMonitor] New block received via zmq  (block_hash=000000000000000000043d68823c5a8e6f5a7f62e4e04456ed4ad87a8b81bce4)
28/02/2021 03:56:07 [Responder] New block received  (block_hash=000000000000000000043d68823c5a8e6f5a7f62e4e04456ed4ad87a8b81bce4, prev_block_hash=0000000000000000000abcd2c9949ffe2363d83f56a041cf31a8dd74af7d9882)
28/02/2021 03:56:07 [Watcher] New block received  (block_hash=000000000000000000043d68823c5a8e6f5a7f62e4e04456ed4ad87a8b81bce4, prev_block_hash=0000000000000000000abcd2c9949ffe2363d83f56a041cf31a8dd74af7d9882)
28/02/2021 03:57:59 [ChainMonitor] New block received via zmq  (block_hash=000000000000000000048bdea371aed1061bf0aea904a3a7f4e5c09e354e6951)
28/02/2021 03:57:59 [Watcher] New block received  (block_hash=000000000000000000048bdea371aed1061bf0aea904a3a7f4e5c09e354e6951, prev_block_hash=000000000000000000043d68823c5a8e6f5a7f62e4e04456ed4ad87a8b81bce4)
28/02/2021 03:57:59 [Responder] New block received  (block_hash=000000000000000000048bdea371aed1061bf0aea904a3a7f4e5c09e354e6951, prev_block_hash=000000000000000000043d68823c5a8e6f5a7f62e4e04456ed4ad87a8b81bce4)
28/02/2021 04:00:12 [ChainMonitor] New block received via zmq  (block_hash=0000000000000000000c1154f8cb93095428a2e9f7c07022539ddc826d239134)
28/02/2021 04:00:12 [Watcher] New block received  (block_hash=0000000000000000000c1154f8cb93095428a2e9f7c07022539ddc826d239134, prev_block_hash=000000000000000000048bdea371aed1061bf0aea904a3a7f4e5c09e354e6951)
28/02/2021 04:00:12 [Responder] New block received  (block_hash=0000000000000000000c1154f8cb93095428a2e9f7c07022539ddc826d239134, prev_block_hash=000000000000000000048bdea371aed1061bf0aea904a3a7f4e5c09e354e6951)
28/02/2021 04:34:39 [ChainMonitor] New block received via zmq  (block_hash=0000000000000000000cd9173e22e0eaaf82621c68d5fd758aa938e23bbf6e65)
28/02/2021 04:34:39 [Responder] New block received  (block_hash=0000000000000000000cd9173e22e0eaaf82621c68d5fd758aa938e23bbf6e65, prev_block_hash=0000000000000000000c1154f8cb93095428a2e9f7c07022539ddc826d239134)
28/02/2021 04:34:39 [Watcher] New block received  (block_hash=0000000000000000000cd9173e22e0eaaf82621c68d5fd758aa938e23bbf6e65, prev_block_hash=0000000000000000000c1154f8cb93095428a2e9f7c07022539ddc826d239134)
28/02/2021 04:49:57 [ChainMonitor] New block received via zmq  (block_hash=000000000000000000033a358a3fa1614e9cb1047638fc5c5acd5ffd047dede2)
28/02/2021 04:49:57 [Watcher] New block received  (block_hash=000000000000000000033a358a3fa1614e9cb1047638fc5c5acd5ffd047dede2, prev_block_hash=0000000000000000000cd9173e22e0eaaf82621c68d5fd758aa938e23bbf6e65)
28/02/2021 04:49:57 [Responder] New block received  (block_hash=000000000000000000033a358a3fa1614e9cb1047638fc5c5acd5ffd047dede2, prev_block_hash=0000000000000000000cd9173e22e0eaaf82621c68d5fd758aa938e23bbf6e65)
28/02/2021 04:53:04 [ChainMonitor] New block received via zmq  (block_hash=0000000000000000000b20cf49764981714c5251c3c58fcd567af6e6b8262596)
28/02/2021 04:53:04 [Watcher] New block received  (block_hash=0000000000000000000b20cf49764981714c5251c3c58fcd567af6e6b8262596, prev_block_hash=000000000000000000033a358a3fa1614e9cb1047638fc5c5acd5ffd047dede2)
28/02/2021 04:53:05 [Responder] New block received  (block_hash=0000000000000000000b20cf49764981714c5251c3c58fcd567af6e6b8262596, prev_block_hash=000000000000000000033a358a3fa1614e9cb1047638fc5c5acd5ffd047dede2)
28/02/2021 05:40:08 [ChainMonitor] New block received via zmq  (block_hash=0000000000000000000003e82cd3154f5e7f9e47431c28e6f2af4f0c93eab28b)
28/02/2021 05:40:08 [Watcher] New block received  (block_hash=0000000000000000000003e82cd3154f5e7f9e47431c28e6f2af4f0c93eab28b, prev_block_hash=0000000000000000000b20cf49764981714c5251c3c58fcd567af6e6b8262596)
28/02/2021 05:40:08 [Responder] New block received  (block_hash=0000000000000000000003e82cd3154f5e7f9e47431c28e6f2af4f0c93eab28b, prev_block_hash=0000000000000000000b20cf49764981714c5251c3c58fcd567af6e6b8262596)
28/02/2021 05:52:43 [ChainMonitor] New block received via zmq  (block_hash=000000000000000000057edf6958bc7afad1e8cc5b432a932a29151d75b82744)
28/02/2021 05:52:43 [Responder] New block received  (block_hash=000000000000000000057edf6958bc7afad1e8cc5b432a932a29151d75b82744, prev_block_hash=0000000000000000000003e82cd3154f5e7f9e47431c28e6f2af4f0c93eab28b)
28/02/2021 05:52:43 [Watcher] New block received  (block_hash=000000000000000000057edf6958bc7afad1e8cc5b432a932a29151d75b82744, prev_block_hash=0000000000000000000003e82cd3154f5e7f9e47431c28e6f2af4f0c93eab28b)
28/02/2021 06:30:25 [ChainMonitor] New block received via zmq  (block_hash=000000000000000000091c2e41bc41f49c2a4fc30dc42ff43d07021c62844af4)
28/02/2021 06:30:25 [Watcher] New block received  (block_hash=000000000000000000091c2e41bc41f49c2a4fc30dc42ff43d07021c62844af4, prev_block_hash=000000000000000000057edf6958bc7afad1e8cc5b432a932a29151d75b82744)
28/02/2021 06:30:25 [Responder] New block received  (block_hash=000000000000000000091c2e41bc41f49c2a4fc30dc42ff43d07021c62844af4, prev_block_hash=000000000000000000057edf6958bc7afad1e8cc5b432a932a29151d75b82744)
28/02/2021 06:40:43 [ChainMonitor] New block received via zmq  (block_hash=00000000000000000009a9562775b694776055fa283489e5ee374205069c3bea)
28/02/2021 06:40:43 [Watcher] New block received  (block_hash=00000000000000000009a9562775b694776055fa283489e5ee374205069c3bea, prev_block_hash=000000000000000000091c2e41bc41f49c2a4fc30dc42ff43d07021c62844af4)
28/02/2021 06:40:43 [Responder] New block received  (block_hash=00000000000000000009a9562775b694776055fa283489e5ee374205069c3bea, prev_block_hash=000000000000000000091c2e41bc41f49c2a4fc30dc42ff43d07021c62844af4)
28/02/2021 06:42:38 [ChainMonitor] New block received via zmq  (block_hash=000000000000000000098a2408160d6d96304869470e79668c91aedbd7d5b292)
28/02/2021 06:42:39 [Responder] New block received  (block_hash=000000000000000000098a2408160d6d96304869470e79668c91aedbd7d5b292, prev_block_hash=00000000000000000009a9562775b694776055fa283489e5ee374205069c3bea)
28/02/2021 06:42:39 [Watcher] New block received  (block_hash=000000000000000000098a2408160d6d96304869470e79668c91aedbd7d5b292, prev_block_hash=00000000000000000009a9562775b694776055fa283489e5ee374205069c3bea)
28/02/2021 06:44:51 [ChainMonitor] New block received via zmq  (block_hash=00000000000000000000446eaefee1ae3c7a02f1a97570253123580a02ce15a1)
28/02/2021 06:44:51 [Responder] New block received  (block_hash=00000000000000000000446eaefee1ae3c7a02f1a97570253123580a02ce15a1, prev_block_hash=000000000000000000098a2408160d6d96304869470e79668c91aedbd7d5b292)
28/02/2021 06:44:51 [Watcher] New block received  (block_hash=00000000000000000000446eaefee1ae3c7a02f1a97570253123580a02ce15a1, prev_block_hash=000000000000000000098a2408160d6d96304869470e79668c91aedbd7d5b292)
28/02/2021 07:05:54 [ChainMonitor] New block received via zmq  (block_hash=0000000000000000000400a19ea3becf6f9e665e195980155f486ee557397e4e)
28/02/2021 07:05:54 [Responder] New block received  (block_hash=0000000000000000000400a19ea3becf6f9e665e195980155f486ee557397e4e, prev_block_hash=00000000000000000000446eaefee1ae3c7a02f1a97570253123580a02ce15a1)
28/02/2021 07:05:54 [Watcher] New block received  (block_hash=0000000000000000000400a19ea3becf6f9e665e195980155f486ee557397e4e, prev_block_hash=00000000000000000000446eaefee1ae3c7a02f1a97570253123580a02ce15a1)
28/02/2021 07:08:24 [ChainMonitor] New block received via zmq  (block_hash=00000000000000000006f1ffe4aa24dce7f239e35a078692b65bf592b4b51949)
28/02/2021 07:08:24 [Responder] New block received  (block_hash=00000000000000000006f1ffe4aa24dce7f239e35a078692b65bf592b4b51949, prev_block_hash=0000000000000000000400a19ea3becf6f9e665e195980155f486ee557397e4e)
28/02/2021 07:08:24 [Watcher] New block received  (block_hash=0

@sr-gi
Copy link
Member

sr-gi commented Mar 3, 2021 •
edited
Loading

Looks like there's some kind of connection problem between the tower and bitcoind. Let me look into it and see if I can figure out what's going on.

Would you mind also providing the deployment setup you used for btcpay in order to reproduce?

@sr-gi
Copy link
Member

sr-gi commented Mar 4, 2021 •
edited
Loading

27/02/2021 16:01:33 [Daemon] Tower identity found. Loading keys
27/02/2021 16:01:33 [Daemon] ERROR: An error occurred: Cannot connect to bitcoind. Shutting down
27/02/2021 16:01:37 [Daemon] Tower identity found. Loading keys
27/02/2021 16:01:41 [Daemon] Tower identity found. Loading keys
27/02/2021 16:01:33 [Daemon] ERROR: An error occurred: Cannot connect to bitcoind. Shutting down

This first bit (I'm assuming the initial bootstrap) happens because bitcoind is not still launched (or at least accesible via RPC), so the teos container is keeping restarting until it gets to connect to bitcoind.

Exception in thread Thread-2:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/threading.py", line 950, in _bootstrap_inner
self.run()
File "/usr/local/lib/python3.9/threading.py", line 888, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.9/site-packages/teos/chain_monitor.py", line 123, in monitor_chain_polling
current_tip = self.block_processor.get_best_block_hash()
File "/usr/local/lib/python3.9/site-packages/teos/block_processor.py", line 62, in get_best_block_hash
block_hash = bitcoin_cli(self.btc_connect_params).getbestblockhash()
File "/usr/local/lib/python3.9/site-packages/teos/utils/auth_proxy.py", line 151, in call
response, status = self._request("POST", self.__url.path, postdata.encode("utf-8"))
File "/usr/local/lib/python3.9/site-packages/teos/utils/auth_proxy.py", line 114, in _request
self.__conn.request(method, path, postdata, headers)
File "/usr/local/lib/python3.9/http/client.py", line 1255, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/lib/python3.9/http/client.py", line 1301, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.9/http/client.py", line 1250, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.9/http/client.py", line 1010, in _send_output
self.send(msg)
File "/usr/local/lib/python3.9/http/client.py", line 950, in send
self.connect()
File "/usr/local/lib/python3.9/http/client.py", line 921, in connect
self.sock = self._create_connection(
File "/usr/local/lib/python3.9/socket.py", line 843, in create_connection
raise err
File "/usr/local/lib/python3.9/socket.py", line 831, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

This second bit happens because of the same reason, but this time while teos is running. Can you check if the bitcoind container is restarting from time to time?

This is related to #19. teos should halt and keep polling bitcoind if the later goes down. bitcoind is really stable though, so it crashing is something quite strange.

@uhlhosting
Copy link
Author

so I am using standard dockerized deployent, all services look normal, none have or report issues. Anything in specific should I provide? I could get you access to the server if you wish, its a testing pre production server, if you think could help debug faster. hence like mentioned this is a clean dockerized version nothing changed.

@sr-gi
Copy link
Member

sr-gi commented Mar 7, 2021

I can give it a look if you give me access. Drop me an email at sergi@talaia.watch

@uhlhosting
Copy link
Author

@sr-gi email sent.

@sr-gi
Copy link
Member

sr-gi commented Mar 9, 2021 •
edited
Loading

@uhlhosting I've been working on a patch for this type of crash. Will test it out on your environment.

#271

@sr-gi
Copy link
Member

sr-gi commented Mar 9, 2021

@uhlhosting

I've been taking a look at the logs and, as I expected, your bitcoind container has been resetting a lot. I say I expected it because that's the first reason that comes to mind for the tower to be unable to connect to bitcoind once it has been bootstrap, given the connection is tested during the bootstrap process and assumed valid from that point onwards. However, this is a rather weird thing, since bitcoind is a really stable software (I'm not that sure to bitcoind docker deployments though, since I'm not that used to them, but I'd expect similar performance).

Every time bitcoind was non-reachable the tower code crashed and therefore the container reset. This issue (the tower crashing if bitcoind was offline) has been known for a while, but it was a rather low priority issue given bitcoind does not tend to crash :S

Here's an example of a crash match:

bitcoind crashes

2021-03-05T14:29:39Z torcontrol thread exit
2021-03-05T14:29:39Z Shutdown: In progress...
2021-03-05T14:29:39Z addcon thread exit
2021-03-05T14:29:39Z net thread exit
2021-03-05T14:29:39Z msghand thread exit
2021-03-05T14:29:43Z opencon thread exit
...2021-03-05T14:29:48Z [default wallet] Releasing wallet
2021-03-05T14:29:49Z Shutdown: done

tower crashes

05/03/2021 14:30:20 [BlockProcessor] ERROR: Couldn't get block hash  (error={'code': -28, 'message': 'Verifying blocks...'})
05/03/2021 14:32:50 [Daemon] Tower identity found. Loading keys
05/03/2021 14:32:52 [Daemon] tower_id = 026247def18440eb3a3e59be829b7c12dd2e86da6989b88644de306132adf72761
05/03/2021 14:32:53 [Daemon] Starting TEOS
05/03/2021 14:32:53 [Daemon] Fresh bootstrap

AFAICT I haven't seen any error on the bitcoind logs that was triggering a reset, so it's either something I'm unaware of, or it was being triggered from outside (docker?). Also, were you doing IBD on that node?

@sr-gi
Copy link
Member

sr-gi commented Mar 9, 2021

Looks like the tower has now been up for around 3 days with no issues.

We could try to patch #271 in and monitor the logs. That won't solve the underlying issue AFAICT (why bitcoind was reseting), but at least would prevent the tower from crashing if it does.

The patch has not been extensively tested, but I think we could go for it given you were running a testing setup.

I'll leave it up to you to decide though.

@uhlhosting
Copy link
Author

Go ahead and patch it. Can fully do what must done to test this.

@sr-gi
Copy link
Member

sr-gi commented Mar 10, 2021

Is the server still up? Tried to ssh and I cannot anymore.

@uhlhosting
Copy link
Author

Hi @sr-gi we have a downtime due to the OVH DC Fire. No Data loss involved, yet this downtime...

@sr-gi
Copy link
Member

sr-gi commented Mar 15, 2021

Oh OK. Can you ping me when the server is back online so I can patch this?

@uhlhosting
Copy link
Author

ofcourse I will do so.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@uhlhosting @sr-gi