elastalert-server docker logs - ERROR elastalert: elastalert: Error finding recent pending alerts: NotFoundError 404

[pratikshatiwari]

Hi,

I have installed the elastalert-server2 and it is running too but in error log i can frequently see below message which is blocking to work elastalert-server & elastalert together.

Installed Python

python 3.9

https://github.com/jertel/elastalert2/
https://github.com/Karql/elastalert2-server

Error log as below:

17:16:42.012Z WARN elastalert: elasticsearch: POST http://localhost:9201/elastalert_status/_search?size=1000 [status:404 request:0.003s]
17:16:42.012Z ERROR elastalert: elasticsearch: POST http://localhost:9201/elastalert_status/_search?size=1000 [status:404 request:0.003s]
17:16:42.013Z ERROR elastalert: elastalert: Error finding recent pending alerts: NotFoundError(404, 'index_not_found_exception', 'no such index [elastalert_status]', elastalert_status, index_or_alias) {'query': {'bool': {'must': {'query_string': {'query': '!exists:aggregate_id AND alert_sent:false'}}, 'filter': {'range': {'alert_time': {'from': '2023-04-09T17:16:42.008720Z', 'to': '2023-04-11T17:16:42.008752Z'}}}}}, 'sort': {'alert_time': {'order': 'asc'}}} Traceback (most recent call last): File "/opt/elastalert/elastalert/elastalert.py", line 1476, in find_recent_pending_alerts res = self.writeback_es.search(index=self.writeback_index, body=query, size=1000) File "/usr/lib/python3.10/site-packages/elasticsearch-7.10.1-py3.10.egg/elasticsearch/client/utils.py", line 152, in _wrapped return func(*args, params=params, headers=headers, **kwargs) File "/usr/lib/python3.10/site-packages/elasticsearch-7.10.1-py3.10.egg/elasticsearch/client/init.py", line 1658, in search return self.transport.perform_request( File "/usr/lib/python3.10/site-packages/elasticsearch-7.10.1-py3.10.egg/elasticsearch/transport.py", line 392, in perform_request raise e File "/usr/lib/python3.10/site-packages/elasticsearch-7.10.1-py3.10.egg/elasticsearch/transport.py", line 358, in perform_request status, headers_response, data = connection.perform_request( File "/usr/lib/python3.10/site-packages/elasticsearch-7.10.1-py3.10.egg/elasticsearch/connection/http_requests.py", line 199, in perform_request self._raise_error(response.status_code, raw_data) File "/usr/lib/python3.10/site-packages/elasticsearch-7.10.1-py3.10.egg/elasticsearch/connection/base.py", line 315, in _raise_error raise HTTP_EXCEPTIONS.get(status_code, TransportError)( elasticsearch.exceptions.NotFoundError: NotFoundError(404, 'index_not_found_exception', 'no such index [elastalert_status]', elastalert_status, index_or_alias)

[Karql]

Hi @pratikshatiwari

ElastAlert2 Server
A server that runs ElastAlert2 and exposes REST API's for manipulating rules and alerts.

Questions related to ElastAlert2 please send directly to its github project: https://github.com/jertel/elastalert2/discussions

But here is the problem:

Error finding recent pending alerts: NotFoundError(404, 'index_not_found_exception', 'no such index [elastalert_status]', elastalert_status, index_or_alias)

Probably something went wrong while executing create index script.

Best regards,
Mateusz

[pratikshatiwari]

Thank you @Karql for your reply i will post the query in jertel discussions forum.

also i am not able to find the issue where is even the create index script clue i am not getting how to check

[Karql]

When you use elastalert2-server create-index should be run automatically: https://github.com/Karql/elastalert2-server/blob/master/src/services/process.service.ts#L53

https://elastalert2.readthedocs.io/en/latest/elastalert_status.html

btw. You should use elastalert2 or elastalert2-server not both.

[pratikshatiwari]

Hello @Karql

Yes also the elastalert indexes are getting created too i verified them through kibnaa & through curl too

curl localhost:9201/_cat/indices/elastalert*
yellow open elastalert_status_status jWJdnw4sTMmgcFQDQKnyeQ 1 1 0 0 225b 225b
yellow open elastalert_status -L13Nm1VTi2iVgvpfjNSug 1 1 0 0 225b 225b
yellow open elastalert_status_past f2JNYQqpSmu6kKaxJFzoCg 1 1 0 0 225b 225b
yellow open elastalert_status_silence IGGSvzL3TqqQXK2PBjj6tw 1 1 0 0 225b 225b
yellow open elastalert_status_error WNCkLyLySsqyzLDNjcO3og 1 1 0 0 225b 225b

Oh is it, but i followed the elastalert-server docker installation method where it also requires the elatalert configuration so accordingly i installed elastalert

docker run -d -p 3030:3030 -p 3333:3333
-v pwd/config/elastalert.yaml:/opt/elastalert/config.yaml
-v pwd/config/elastalert-test.yaml:/opt/elastalert/config-test.yaml
-v pwd/config/config.json:/opt/elastalert-server/config/config.json
-v pwd/rules:/opt/elastalert/rules
-v pwd/rule_templates:/opt/elastalert/rule_templates
--net="host"
--name elastalert2-server karql/elastalert2-server:latest

my end goal is to manage the email alert rules using kibana plugin

[Karql]

Here you can find working example: https://github.com/Karql/elastalert-kibana-plugin/tree/master/dev/test-env
You only need to download plugin https://github.com/Karql/elastalert-kibana-plugin/releases/download/1.6.1/elastalertKibanaPlugin-1.6.1-8.6.2.zip and place it in kibana/plugins/elastalertKibanaPlugin-8.6.2.zip

This also can be useful: https://github.com/Karql/elastalert-kibana-plugin/blob/master/CONTRIBUTING.md