Elastalert2 Docker to Opensearch Docker connection issue #1401

wmashal · 2024-03-25T11:53:01Z

wmashal
Mar 25, 2024

Hi

I am trying to run a POC using opensearch docker and elastalert2 docker
But still face issue with communication between the dockers, when elastalert2 trying to connect to opensearch

My deployment steps:

For open search I am using the official docker image https://hub.docker.com/r/opensearchproject/opensearch, when I am testing the opensearc using curl it works

I CHANGED the PASSWORD WITH STARS

docker run -d -p 9200:9200 -p 9600:9600 -e OPENSEARCH_INITIAL_ADMIN_PASSWORD=******* -e "discovery.type=single-node"  --name opensearch-node opensearchproject/opensearch:latest

I am using the elastalert2 sample in the documentation "Geting Started" https://elastalert2.readthedocs.io/en/latest/running_elastalert.html

My elastalert2 config is the following

rules_folder: /opt/elastalert/rules

run_every:
  seconds: 10

buffer_time:
  minutes: 15

es_host: localhost
es_port: 9200
#use_ssl: True
writeback_index: elastalert_status

alert_time_limit:
  days: 2

I got the following error when I run the docker

WARNING:elasticsearch:GET http://localhost:9200/ [status:N/A request:0.000s]
2024-03-25 13:48:46 Traceback (most recent call last):
2024-03-25 13:48:46   File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 174, in _new_conn
2024-03-25 13:48:46     conn = connection.create_connection(
2024-03-25 13:48:46            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-25 13:48:46   File "/usr/local/lib/python3.12/site-packages/urllib3/util/connection.py", line 95, in create_connection
2024-03-25 13:48:46     raise err
2024-03-25 13:48:46   File "/usr/local/lib/python3.12/site-packages/urllib3/util/connection.py", line 85, in create_connection
2024-03-25 13:48:46     sock.connect(sa)
2024-03-25 13:48:46 ConnectionRefusedError: [Errno 111] Connection refused
2024-03-25 13:48:46 
2024-03-25 13:48:46 During handling of the above exception, another exception occurred:
2024-03-25 13:48:46 
2024-03-25 13:48:46 Traceback (most recent call last):
2024-03-25 13:48:46   File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 715, in urlopen
2024-03-25 13:48:46     httplib_response = self._make_request(
2024-03-25 13:48:46                        ^^^^^^^^^^^^^^^^^^^
2024-03-25 13:48:46   File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 416, in _make_request
2024-03-25 13:48:46     conn.request(method, url, **httplib_request_kw)
2024-03-25 13:48:46   File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 244, in request
2024-03-25 13:48:46     super(HTTPConnection, self).request(method, url, body=body, headers=headers)
2024-03-25 13:48:46   File "/usr/local/lib/python3.12/http/client.py", line 1331, in request
2024-03-25 13:48:46     self._send_request(method, url, body, headers, encode_chunked)
2024-03-25 13:48:46   File "/usr/local/lib/python3.12/http/client.py", line 1377, in _send_request
2024-03-25 13:48:46     self.endheaders(body, encode_chunked=encode_chunked)
2024-03-25 13:48:46   File "/usr/local/lib/python3.12/http/client.py", line 1326, in endheaders
2024-03-25 13:48:46     self._send_output(message_body, encode_chunked=encode_chunked)
2024-03-25 13:48:46   File "/usr/local/lib/python3.12/http/client.py", line 1085, in _send_output
2024-03-25 13:48:46     self.send(msg)
2024-03-25 13:48:46   File "/usr/local/lib/python3.12/http/client.py", line 1029, in send
2024-03-25 13:48:46     self.connect()
2024-03-25 13:48:46   File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 205, in connect
2024-03-25 13:48:46     conn = self._new_conn()
2024-03-25 13:48:46            ^^^^^^^^^^^^^^^^
2024-03-25 13:48:46   File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 186, in _new_conn
2024-03-25 13:48:46     raise NewConnectionError(
2024-03-25 13:48:46 urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0xffffa0379fa0>: Failed to establish a new connection: [Errno 111] Connection refused
2024-03-25 13:48:46 
2024-03-25 13:48:46 During handling of the above exception, another exception occurred:
2024-03-25 13:48:46 
2024-03-25 13:48:46 Traceback (most recent call last):
2024-03-25 13:48:46   File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 486, in send
2024-03-25 13:48:46     resp = conn.urlopen(
2024-03-25 13:48:46            ^^^^^^^^^^^^^
2024-03-25 13:48:46   File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 799, in urlopen
2024-03-25 13:48:46     retries = retries.increment(
2024-03-25 13:48:46               ^^^^^^^^^^^^^^^^^^
2024-03-25 13:48:46   File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 592, in increment
2024-03-25 13:48:46     raise MaxRetryError(_pool, url, error or ResponseError(cause))
2024-03-25 13:48:46 urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=9200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xffffa0379fa0>: Failed to establish a new connection: [Errno 111] Connection refused'))
2024-03-25 13:48:46 
2024-03-25 13:48:46 During handling of the above exception, another exception occurred:
2024-03-25 13:48:46 
2024-03-25 13:48:46 Traceback (most recent call last):
2024-03-25 13:48:46   File "/usr/local/lib/python3.12/site-packages/elasticsearch/connection/http_requests.py", line 161, in perform_request
2024-03-25 13:48:46     response = self.session.send(prepared_request, **send_kwargs)
2024-03-25 13:48:46                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-25 13:48:46   File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
2024-03-25 13:48:46     r = adapter.send(request, **kwargs)
2024-03-25 13:48:46         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-25 13:48:46   File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 519, in send
2024-03-25 13:48:46     raise ConnectionError(e, request=request)
2024-03-25 13:48:46 requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=9200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xffffa0379fa0>: Failed to establish a new connection: [Errno 111] Connection refused'))
2024-03-25 13:48:46 WARNING:elasticsearch:GET http://localhost:9200/ [status:N/A request:0.000s]
2024-03-25 13:48:46 Traceback (most recent call last):
2024-03-25 13:48:46   File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 174, in _new_conn
2024-03-25 13:48:46     conn = connection.create_connection(
2024-03-25 13:48:46            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-25 13:48:46   File "/usr/local/lib/python3.12/site-packages/urllib3/util/connection.py", line 95, in create_connection
2024-03-25 13:48:46     raise err
2024-03-25 13:48:46   File "/usr/local/lib/python3.12/site-packages/urllib3/util/connection.py", line 85, in create_connection
2024-03-25 13:48:46     sock.connect(sa)
2024-03-25 13:48:46 ConnectionRefusedError: [Errno 111] Connection refused
2024-03-25 13:48:46 
2024-03-25 13:48:46 During handling of the above exception, another exception occurred:
2024-03-25 13:48:46 
2024-03-25 13:48:46 Traceback (most recent call last):
2024-03-25 13:48:46   File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 715, in urlopen
2024-03-25 13:48:46     httplib_response = self._make_request(
2024-03-25 13:48:46                        ^^^^^^^^^^^^^^^^^^^
2024-03-25 13:48:46   File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 416, in _make_request
2024-03-25 13:48:46     conn.request(method, url, **httplib_request_kw)
2024-03-25 13:48:46   File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 244, in request
2024-03-25 13:48:46     super(HTTPConnection, self).request(method, url, body=body, headers=headers)
2024-03-25 13:48:46   File "/usr/local/lib/python3.12/http/client.py", line 1331, in request
2024-03-25 13:48:46     self._send_request(method, url, body, headers, encode_chunked)
2024-03-25 13:48:46   File "/usr/local/lib/python3.12/http/client.py", line 1377, in _send_request
2024-03-25 13:48:46     self.endheaders(body, encode_chunked=encode_chunked)
2024-03-25 13:48:46   File "/usr/local/lib/python3.12/http/client.py", line 1326, in endheaders
2024-03-25 13:48:46     self._send_output(message_body, encode_chunked=encode_chunked)
2024-03-25 13:48:46   File "/usr/local/lib/python3.12/http/client.py", line 1085, in _send_output
2024-03-25 13:48:46     self.send(msg)
2024-03-25 13:48:46   File "/usr/local/lib/python3.12/http/client.py", line 1029, in send
2024-03-25 13:48:46     self.connect()
2024-03-25 13:48:46   File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 205, in connect
2024-03-25 13:48:46     conn = self._new_conn()
2024-03-25 13:48:46            ^^^^^^^^^^^^^^^^
2024-03-25 13:48:46   File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 186, in _new_conn
2024-03-25 13:48:46     raise NewConnectionError(
2024-03-25 13:48:46 urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0xffff9fdcfe60>: Failed to establish a new connection: [Errno 111] Connection refused
2024-03-25 13:48:46 
2024-03-25 13:48:46 During handling of the above exception, another exception occurred:
2024-03-25 13:48:46 
2024-03-25 13:48:46 Traceback (most recent call last):
2024-03-25 13:48:46   File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 486, in send
2024-03-25 13:48:46     resp = conn.urlopen(
2024-03-25 13:48:46            ^^^^^^^^^^^^^
2024-03-25 13:48:46   File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 799, in urlopen
2024-03-25 13:48:46     retries = retries.increment(
2024-03-25 13:48:46               ^^^^^^^^^^^^^^^^^^
2024-03-25 13:48:46   File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 592, in increment
2024-03-25 13:48:46     raise MaxRetryError(_pool, url, error or ResponseError(cause))
2024-03-25 13:48:46 urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=9200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xffff9fdcfe60>: Failed to establish a new connection: [Errno 111] Connection refused'))

Important notes:

I tried to enable #use_ssl: True but I got certificate error
I add new bride network and I connect the two dockers to it not help also

My docker run commands

Thanks

Answered by wmashal

Mar 26, 2024

@jertel You right I rerun the opensearch docker by setting plugins.security.disabled=true now its reachable

View full answer

jertel · 2024-03-25T12:06:25Z

jertel
Mar 25, 2024
Maintainer

Try setting es_host to opensearch-node. As long as both containers are running on the same docker network that should route correctly.

5 replies

jertel Mar 25, 2024
Maintainer

You can try adding a --link opensearch-node:opensearch-node to your ElastAlert 2 docker run command line, to force it to create a network link to the Opensearch container.

On a side note, this is really getting into Docker networking fundamentals, which this discussion forum is not intended to support.

wmashal Mar 25, 2024
Author

@jertel same issue , regarding the "Docker networking fundamentals" I know that but I thought its a bug in elastalert2

I tried also with the following configurations

es_host: opensearch-node
es_port: 9200
use_ssl: True
verify_certs: False

but got error 401

 WARNING:py.warnings:/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'opensearch-node'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
2024-03-25 14:53:16   warnings.warn(
2024-03-25 14:53:16 
2024-03-25 14:53:16 WARNING:elasticsearch:GET https://opensearch-node:9200/ [status:401 request:0.008s]
2024-03-25 14:53:16 Traceback (most recent call last):
2024-03-25 14:53:16   File "/usr/local/bin/elastalert-create-index", line 8, in <module>
2024-03-25 14:53:16     sys.exit(main())
2024-03-25 14:53:16              ^^^^^^
2024-03-25 14:53:16   File "/usr/local/lib/python3.12/site-packages/elastalert/create_index.py", line 242, in main
2024-03-25 14:53:16     create_index_mappings(es_client=es, ea_index=index, recreate=args.recreate, old_ea_index=old_index)
2024-03-25 14:53:16   File "/usr/local/lib/python3.12/site-packages/elastalert/create_index.py", line 24, in create_index_mappings
2024-03-25 14:53:16     esversion = get_version_from_cluster_info(es_client)
2024-03-25 14:53:16                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-25 14:53:16   File "/usr/local/lib/python3.12/site-packages/elastalert/util.py", line 597, in get_version_from_cluster_info
2024-03-25 14:53:16     esinfo = client.info()['version']
2024-03-25 14:53:16              ^^^^^^^^^^^^^
2024-03-25 14:53:16   File "/usr/local/lib/python3.12/site-packages/elasticsearch/client/utils.py", line 152, in _wrapped
2024-03-25 14:53:16     return func(*args, params=params, headers=headers, **kwargs)
2024-03-25 14:53:16            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-25 14:53:16   File "/usr/local/lib/python3.12/site-packages/elasticsearch/client/__init__.py", line 286, in info
2024-03-25 14:53:16     return self.transport.perform_request(
2024-03-25 14:53:16            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-25 14:53:16   File "/usr/local/lib/python3.12/site-packages/elasticsearch/transport.py", line 392, in perform_request
2024-03-25 14:53:16     raise e
2024-03-25 14:53:16   File "/usr/local/lib/python3.12/site-packages/elasticsearch/transport.py", line 358, in perform_request
2024-03-25 14:53:16     status, headers_response, data = connection.perform_request(
2024-03-25 14:53:16                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-25 14:53:16   File "/usr/local/lib/python3.12/site-packages/elasticsearch/connection/http_requests.py", line 199, in perform_request
2024-03-25 14:53:16     self._raise_error(response.status_code, raw_data)
2024-03-25 14:53:16   File "/usr/local/lib/python3.12/site-packages/elasticsearch/connection/base.py", line 315, in _raise_error
2024-03-25 14:53:16     raise HTTP_EXCEPTIONS.get(status_code, TransportError)(
2024-03-25 14:53:16 elasticsearch.exceptions.AuthenticationException: AuthenticationException(401, '')
2024-03-25 14:53:17 WARNING:py.warnings:/usr/local/lib/python3.12/site-packages/elasticsearch/connection/http_requests.py:134: UserWarning: Connecting to https://opensearch-node:9200 using SSL with verify_certs=False is insecure.
2024-03-25 14:53:17   warnings.warn(
2024-03-25 14:53:17 
2024-03-25 14:53:17 WARNING:py.warnings:/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'opensearch-node'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
2024-03-25 14:53:17   warnings.warn(
2024-03-25 14:53:17 
2024-03-25 14:53:17 WARNING:elasticsearch:GET https://opensearch-node:9200/ [status:401 request:0.021s]
2024-03-25 14:53:17 WARNING:elastalert:Failed to retrieve cluster version information, retrying in 3 seconds
2024-03-25 14:53:20 WARNING:py.warnings:/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'opensearch-node'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
2024-03-25 14:53:20   warnings.warn(
2024-03-25 14:53:20 
2024-03-25 14:53:20 WARNING:elasticsearch:GET https://opensearch-node:9200/ [status:401 request:0.008s]
2024-03-25 14:53:20 WARNING:elastalert:Failed to retrieve cluster version information, retrying in 3 seconds
2024-03-25 14:53:23 WARNING:py.warnings:/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'opensearch-node'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
2024-03-25 14:53:23   warnings.warn(
2024-03-25 14:53:23 
2024-03-25 14:53:23 WARNING:elasticsearch:GET https://opensearch-node:9200/ [status:401 request:0.006s]
2024-03-25 14:53:23 Traceback (most recent call last):
2024-03-25 14:53:23   File "/usr/local/bin/elastalert-create-index", line 8, in <module>
2024-03-25 14:53:23     sys.exit(main())
2024-03-25 14:53:23              ^^^^^^
2024-03-25 14:53:23   File "/usr/local/lib/python3.12/site-packages/elastalert/create_index.py", line 242, in main
2024-03-25 14:53:23     create_index_mappings(es_client=es, ea_index=index, recreate=args.recreate, old_ea_index=old_index)
2024-03-25 14:53:23   File "/usr/local/lib/python3.12/site-packages/elastalert/create_index.py", line 24, in create_index_mappings
2024-03-25 14:53:23     esversion = get_version_from_cluster_info(es_client)
2024-03-25 14:53:23                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-25 14:53:23   File "/usr/local/lib/python3.12/site-packages/elastalert/util.py", line 597, in get_version_from_cluster_info
2024-03-25 14:53:23     esinfo = client.info()['version']
2024-03-25 14:53:23              ^^^^^^^^^^^^^
2024-03-25 14:53:23   File "/usr/local/lib/python3.12/site-packages/elasticsearch/client/utils.py", line 152, in _wrapped
2024-03-25 14:53:23     return func(*args, params=params, headers=headers, **kwargs)
2024-03-25 14:53:23            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-25 14:53:23   File "/usr/local/lib/python3.12/site-packages/elasticsearch/client/__init__.py", line 286, in info
2024-03-25 14:53:23     return self.transport.perform_request(
2024-03-25 14:53:23            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-25 14:53:23   File "/usr/local/lib/python3.12/site-packages/elasticsearch/transport.py", line 392, in perform_request
2024-03-25 14:53:23     raise e
2024-03-25 14:53:23   File "/usr/local/lib/python3.12/site-packages/elasticsearch/transport.py", line 358, in perform_request
2024-03-25 14:53:23     status, headers_response, data = connection.perform_request(
2024-03-25 14:53:23                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-25 14:53:23   File "/usr/local/lib/python3.12/site-packages/elasticsearch/connection/http_requests.py", line 199, in perform_request
2024-03-25 14:53:23     self._raise_error(response.status_code, raw_data)
2024-03-25 14:53:23   File "/usr/local/lib/python3.12/site-packages/elasticsearch/connection/base.py", line 315, in _raise_error
2024-03-25 14:53:23     raise HTTP_EXCEPTIONS.get(status_code, TransportError)(
2024-03-25 14:53:23 elasticsearch.exceptions.AuthenticationException: AuthenticationException(401, '')

jertel Mar 25, 2024
Maintainer

No, it is not a bug in ElastAlert 2. You have networking and configuration issues with your specific deployment.

A 401 error means it is now able to reach OpenSearch but cannot authenticate, probably due to incorrect credentials provided to ElastAlert 2.

wmashal Mar 26, 2024
Author

@jertel You right I rerun the opensearch docker by setting plugins.security.disabled=true now its reachable

Answer selected by wmashal

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Elastalert2 Docker to Opensearch Docker connection issue #1401

{{title}}

Replies: 1 comment 5 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Elastalert2 Docker to Opensearch Docker connection issue #1401

wmashal Mar 25, 2024

Replies: 1 comment · 5 replies

jertel Mar 25, 2024 Maintainer

jertel Mar 25, 2024 Maintainer

wmashal Mar 25, 2024 Author

jertel Mar 25, 2024 Maintainer

wmashal Mar 26, 2024 Author

wmashal
Mar 25, 2024

Replies: 1 comment 5 replies

jertel
Mar 25, 2024
Maintainer

jertel Mar 25, 2024
Maintainer

wmashal Mar 25, 2024
Author

jertel Mar 25, 2024
Maintainer

wmashal Mar 26, 2024
Author