Alert for Future Logs

[albertoCrego]

Hello, we have an issue with our OpenSearch Cluster which creates data in the future index, to detect it we want to create an Elastalert that notifies us if there are documents with a timestamp in the future.

I tested alerts like this:

alert_subject: There are future indices.
type: any

index: logs-*
num_events: 1
filter:
- query:
    query_string:
        query: "@timestamp:>now"
use_count_query: true

But testing the alert only scrapes information for 5m and my dummy document is in 4 days.

elastalert-test-rule rules/opensearch-future-indices.yaml
...
...
INFO:elastalert:Queried rule rules/opensearch-future-indices from 2024-03-21 08:30 UTC to 2024-03-21 08:35 UTC: 0 hits

Is there any option to scrape information from the "future"?
I already test the timeframe, the buffer, etc but everything looks related to the past.

[jertel]

I've not attempted to do what you're asking, but you could try using a negative query_delay.

Ex:

query_delay: 
  days: -7

[albertoCrego]

Seems that the maximum query delay is 24h:

INFO:elastalert:Queried rule /tmp/opensearch-acrego7 from 2024-03-24 09:18 UTC to 2024-03-25 09:18 UTC: 0 / 0 hits

With the following configuration:

type: any


index: logs-preprod-services-*
num_events: 1
filter:
- query:
    query_string:
        query: “@timestamp:>now”

limit_execution: “* 8-17 * * 1-5"
query_delay:
  days: -7
buffer_time:
  days: 1

I already tested without buffer_time and same result but in 15m buckets.

[jertel]

The other alternative that might work for you is to set the explicit state/end date range on the command line.

--rule my_rule.yaml --start 2024-03-25T00:00:00.000Z --end 2024-04-03T00:00:00.000Z

[albertoCrego]

The problem is today we want to monitor the next week, but periodically... I mean we want to detect this in the future, not only now.

[jertel]

Understood. I circled back to the first option and found that my initial run did search 7 days into the future. So I don't know why you're seeing a 24h cap on the query_delay. I used the following to test:

query_delay:
  days: -7
buffer_time:
  hours: 1

This resulted in a large number of queries, starting with this timerange

2024-03-25 08:17:05,342     INFO           elastalert Queried rule any_test39 from 2024-03-25 09:14 EDT to 2024-03-25 10:14 EDT: 0 / 0 hits

and ending with this

2024-03-25 08:17:06,018     INFO           elastalert Queried rule any_test39 from 2024-04-01 08:14 EDT to 2024-04-01 08:17 EDT: 0 / 0 hits

I did have to give it a --start 2024-03-25T12:14:00.000Z to force it to start now, instead of using its default logic which is to start at the end_time - buffer_time. But once it was off and running it would keep going.

The downside of this option is that when it finishes that initial 7 day search it will not search again until that 7 day period elapses.

Based on this behavior the current implementation of ElastAlert 2 may not be suitable for this use case. This project was designed to search historic data, up to the current time. Adjusting it to handle queries for data in the future is likely going to require a decent investment of someone's time.

[albertoCrego]

The type is any in your alert?

cat /tmp/opensearch-test1.yaml
type: any

index: logs-preprod-services-*
num_events: 1
filter:
- query:
    query_string:
        query: “@timestamp:>now”

query_delay:
  days: -7
buffer_time:
  hours: 1
...
elastalert@elastalert-8674c56645-5fgkz:~$ elastalert-test-rule /tmp/opensearch-test1.yaml
...
INFO:apscheduler.scheduler:Adding job tentatively -- it will be properly scheduled when the scheduler starts
INFO:elastalert:Queried rule /tmp/opensearch-acrego-test1 from 2024-03-25 11:38 UTC to 2024-03-25 12:38 UTC: 0 / 0 hits
...
...
INFO:elastalert:Queried rule /tmp/opensearch-acrego-test1 from 2024-03-26 10:38 UTC to 2024-03-26 11:38 UTC: 0 / 0 hits

[jertel]

Yes.

Rule:

name: any_test39
type: any

index: "*"
query_delay:
  days: -7
buffer_time:
  hours: 1

alert: debug

and config.yaml:

rules_folder: rules

run_every:
  seconds: 10

realert:
  minutes: 1
  
buffer_time:
  minutes: 60
...
elastic settings, etc follow

[albertoCrego]

I should do something different than you because I don't understand what because I'm still getting logs from the last 24 hours 🤔

opensearch-test.yaml

name: any_test39
type: any
index: "logs-preprod-services-*"

filter:
- query:
    query_string:
        query: "@timestamp:>now"

query_delay:
  days: -7
buffer_time:
  hours: 1

alert: debug

config.yaml

rules_folder: /opt/elastalert/rules
scan_subdirectories: true
run_every:
  minutes: 5
realert:
  minutes: 60
buffer_time:
  minutes: 15
 ....
 alerting config 
 ....

Run Commands

elastalert-test-rule /tmp/opensearch-test.yaml --config=config.yaml
...
ommited
...
INFO:elastalert:Queried rule any_test39 from 2024-03-25 15:36 UTC to 2024-03-25 16:36 UTC: 0 / 0 hits
INFO:elastalert:Queried rule any_test39 from 2024-03-25 16:36 UTC to 2024-03-25 17:36 UTC: 0 / 0 hits
...
ommited
...
INFO:elastalert:Queried rule any_test39 from 2024-03-26 13:36 UTC to 2024-03-26 14:36 UTC: 0 / 0 hits
INFO:elastalert:Queried rule any_test39 from 2024-03-26 14:36 UTC to 2024-03-26 15:36 UTC: 0 / 0 hits

Would have written the following documents to writeback index (default is elastalert_status):

elastalert_status - {'rule_name': 'any_test39', 'endtime': datetime.datetime(2024, 3, 26, 15, 36, 18, 220041, tzinfo=tzutc()), 'starttime': datetime.datetime(2024, 3, 25, 15, 36, 18, 220041, tzinfo=tzutc()), 'matches': 0, 'hits': 0, '@timestamp': datetime.datetime(2024, 3, 26, 15, 36, 20, 106050, tzinfo=tzutc()), 'time_taken': 1.510528326034546}

It's strange 🤔 can you see any difference?

[jertel]

I don't see anything. Try running it with debug logging. Perhaps you'll see something that gives you a clue.