Comparing values percentually

[lufimoreira94]

Hi. I'm trying to create a rule that triggers when a duration is 150% above the average time of the last 10 minutes, grouped by an Id. I created an elasticsearch transform to facilitate this process, in which it returns that average time grouped by said Id.
What I'm missing is how to compare those values percentually.

Here's a simple example in case I didn't explain it well:
Imagine that in the last 10 minutes there are 5 logs where 'labels.db_statement_encoded' is defined as "1234", and each duration is, respectively, 2, 4, 3, 2, 4. Then, the average duration would be 3. After that, another log with labels.db_statement_encoded equal to "1234" comes with a duration of 4. That should not trigger the rule, since 4 is lower than 150% of the previous average. But if the duration were to be 6, then it would trigger, since it would be 200% of the average duration of the last 10 minutes.

Long story short, I need to know when X number is 150% or more the value of Y.

Thanks in advance!

[jertel]

Perhaps the Spike Aggregation rule type will solve your needs. Documentation is online at https://elastalert2.readthedocs.io/en/latest/ruletypes.html#spike-aggregation.

[lufimoreira94]

Thanks, that sounds right. I created this rule:

`name: "Under Performance Error Alert v2"
type: spike_aggregation
is_enabled: true

index: ".ds-traces-apm-default-*"

filter:

• term:
  processor.event: "span"
• term:
  span.type: "db"

metric_agg_key: span.duration.us
metric_agg_type: avg
query_key: labels.db_statement_encoded

spike_height: 1
spike_type: up

threshold_cur: 1
#min_doc_count: 5

timeframe:
minutes: 1
buffer_time:
minutes: 1

alert:

• "post"

http_post_url: "https://dsads.wiremockapi.cloud/"
http_post_headers:
Content-Type: "application/json"`

This is for the sake of testing, however, I get this error:
elastalert2-1 | 2025-01-13 14:11:34,737 ERROR Traceback (most recent call last): elastalert2-1 | File "/usr/local/lib/python3.13/site-packages/elastalert/elastalert.py", line 1268, in handle_rule_execution elastalert2-1 | num_matches = self.run_rule(rule, endtime, rule.get('initial_starttime')) elastalert2-1 | File "/usr/local/lib/python3.13/site-packages/elastalert/elastalert.py", line 887, in run_rule elastalert2-1 | rule['type'].garbage_collect(tmp_endtime) elastalert2-1 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^ elastalert2-1 | File "/usr/local/lib/python3.13/site-packages/elastalert/ruletypes.py", line 576, in garbage_collect elastalert2-1 | self.handle_event(placeholder, 0, qk) elastalert2-1 | ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ elastalert2-1 | File "/usr/local/lib/python3.13/site-packages/elastalert/ruletypes.py", line 500, in handle_event elastalert2-1 | if self.find_matches(ref, cur): elastalert2-1 | ~~~~~~~~~~~~~~~~~^^^^^^^^^^ elastalert2-1 | File "/usr/local/lib/python3.13/site-packages/elastalert/ruletypes.py", line 527, in find_matches elastalert2-1 | if (cur < self.rules.get('threshold_cur', 0) or elastalert2-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ elastalert2-1 | TypeError: '<' not supported between instances of 'NoneType' and 'int'

It seems like the current values aren't being properly loaded. Am I missing something?

[jertel]

Do some timerange windows not have any events?

[lufimoreira94]

Yes, and the chance is higher since I'm using query_key I'm assuming. The problem is that I think that error is disabling my rule
elastalert2-1 | 2025-01-14 09:45:57,093 INFO Disabled rules are: ['Under Performance Error Alert v2']
I'm not sure though

[jertel]

Ok, try specifying disable_rules_on_error: false and let's see if the rule behaves as expected, aside from the error when there's no data.

[lufimoreira94]

It worked. Thanks mate!

[lufimoreira94]

Ignore that last message I deleted, it turned out to be history messages that, since the response was 404, didn't mark them as sent, so it kept sending them. I just got confused because of the timespans.

However, I'm not getting any results anymore, even tho I changed spike_height to 1 and spike_type to both. The only change are the timeframe minutes that I updated to 5, besides that, the rule mantains the same structure.

Here is some debug information:
elastalert2-1 | 2025-01-23 11:48:16,542 DEBUG Looking for jobs to run
elastalert2-1 | 2025-01-23 11:48:16,542 DEBUG Next wakeup is due at 2025-01-23 11:48:47.989822+00:00 (in 31.446945 seconds)
elastalert2-1 | 2025-01-23 11:48:16,543 INFO Running job "Internal: Handle Config Change (trigger: interval[0:01:00], next run at: 2025-01-23 11:49:16 UTC)" (scheduled at 2025-01-23 11:48:16.541782+00:00)
elastalert2-1 | 2025-01-23 11:48:16,543 INFO Running job "Internal: Handle Pending Alerts (trigger: interval[0:01:00], next run at: 2025-01-23 11:49:16 UTC)" (scheduled at 2025-01-23 11:48:16.541439+00:00)
elastalert2-1 | 2025-01-23 11:48:16,549 DEBUG http://172.17.0.1:9200 "POST /elastalert_status/_search?size=1000 HTTP/1.1" 200 None
elastalert2-1 | 2025-01-23 11:48:16,550 INFO Background alerts thread 0 pending alerts sent at 2025-01-23 11:48 UTC
elastalert2-1 | 2025-01-23 11:48:16,550 INFO Job "Internal: Handle Pending Alerts (trigger: interval[0:01:00], next run at: 2025-01-23 11:49:16 UTC)" executed successfully
elastalert2-1 | 2025-01-23 11:48:16,551 INFO Background configuration change check run at 2025-01-23 11:48 UTC
elastalert2-1 | 2025-01-23 11:48:16,551 INFO Job "Internal: Handle Config Change (trigger: interval[0:01:00], next run at: 2025-01-23 11:49:16 UTC)" executed successfully
elastalert2-1 | 2025-01-23 11:48:16,556 INFO Disabled rules are: []
elastalert2-1 | 2025-01-23 11:48:16,556 INFO Sleeping for 59.999792 seconds

elastalert2-1 | 2025-01-23 11:48:47,990 DEBUG Looking for jobs to run
elastalert2-1 | 2025-01-23 11:48:47,991 DEBUG Next wakeup is due at 2025-01-23 11:49:16.541439+00:00 (in 28.550339 seconds)
elastalert2-1 | 2025-01-23 11:48:47,991 INFO Running job "Rule: Under Performance Error Alert v2 (trigger: interval[0:01:00], next run at: 2025-01-23 11:49:51 UTC)" (scheduled at 2025-01-23 11:48:47.989822+00:00)
elastalert2-1 | 2025-01-23 11:48:48,011 DEBUG http://172.17.0.1:9200 "POST /.ds-traces-apm-default-*/_search?ignore_unavailable=true&size=0 HTTP/1.1" 200 None
elastalert2-1 | 2025-01-23 11:48:48,034 DEBUG http://172.17.0.1:9200 "POST /elastalert_status_status/_doc HTTP/1.1" 201 180
elastalert2-1 | 2025-01-23 11:48:48,035 INFO Ran Under Performance Error Alert v2 from 2025-01-23 11:46 UTC to 2025-01-23 11:48 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
elastalert2-1 | 2025-01-23 11:48:48,035 INFO Under Performance Error Alert v2 range 118
elastalert2-1 | 2025-01-23 11:48:48,035 INFO Job "Rule: Under Performance Error Alert v2 (trigger: interval[0:01:00], next run at: 2025-01-23 11:49:51 UTC)" executed successfully

I apologize for being a bore.

[lufimoreira94]

I am also not sure if this is working properly everytime. When I have multiple calls with different query_keys, I suspect that, if one of them does not have something to compare to, it breaks all the cycle, not sending any alerts about anything, even if another query_key was compatible with the rule.

My suspicion derives from the log, that does not give me a similiar line to this:
elastalert2-1 | 2025-01-23 11:48:48,035 INFO Ran Under Performance Error Alert v2 from 2025-01-23 11:46 UTC to 2025-01-23 11:48 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent

Here are the logs when the error occurs
elastalert2-1 | 2025-01-23 12:13:57,603 DEBUG Looking for jobs to run
elastalert2-1 | 2025-01-23 12:13:57,604 DEBUG Next wakeup is due at 2025-01-23 12:14:16.541439+00:00 (in 18.937097 seconds)
elastalert2-1 | 2025-01-23 12:13:57,604 INFO Running job "Rule: Under Performance Error Alert v2 (trigger: interval[0:01:00], next run at: 2025-01-23 12:15:02 UTC)" (scheduled at 2025-01-23 12:13:57.600723+00:00)
elastalert2-1 | 2025-01-23 12:13:57,782 DEBUG http://172.17.0.1:9200 "POST /.ds-traces-apm-default-/_search?ignore_unavailable=true&size=0 HTTP/1.1" 200 None
elastalert2-1 | 2025-01-23 12:13:57,816 DEBUG http://172.17.0.1:9200 "POST /.ds-traces-apm-default-/_search?ignore_unavailable=true&size=0 HTTP/1.1" 200 None
elastalert2-1 | 2025-01-23 12:13:57,822 ERROR Traceback (most recent call last):
elastalert2-1 | File "/usr/local/lib/python3.13/site-packages/elastalert/elastalert.py", line 1268, in handle_rule_execution
elastalert2-1 | num_matches = self.run_rule(rule, endtime, rule.get('initial_starttime'))
elastalert2-1 | File "/usr/local/lib/python3.13/site-packages/elastalert/elastalert.py", line 887, in run_rule
elastalert2-1 | rule['type'].garbage_collect(tmp_endtime)
elastalert2-1 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
elastalert2-1 | File "/usr/local/lib/python3.13/site-packages/elastalert/ruletypes.py", line 576, in garbage_collect
elastalert2-1 | self.handle_event(placeholder, 0, qk)
elastalert2-1 | ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
elastalert2-1 | File "/usr/local/lib/python3.13/site-packages/elastalert/ruletypes.py", line 500, in handle_event
elastalert2-1 | if self.find_matches(ref, cur):
elastalert2-1 | ~~~~~~~~~~~~~~~~~^^^^^^^^^^
elastalert2-1 | File "/usr/local/lib/python3.13/site-packages/elastalert/ruletypes.py", line 527, in find_matches
elastalert2-1 | if (cur < self.rules.get('threshold_cur', 0) or
elastalert2-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
elastalert2-1 | TypeError: '<' not supported between instances of 'NoneType' and 'int'
elastalert2-1 |
elastalert2-1 | 2025-01-23 12:13:57,823 ERROR Uncaught exception running rule Under Performance Error Alert v2: '<' not supported between instances of 'NoneType' and 'int'
elastalert2-1 | 2025-01-23 12:13:57,856 DEBUG http://172.17.0.1:9200 "POST /elastalert_status_error/_doc HTTP/1.1" 201 179
elastalert2-1 | 2025-01-23 12:13:57,857 INFO Job "Rule: Under Performance Error Alert v2 (trigger: interval[0:01:00], next run at: 2025-01-23 12:15:02 UTC)" executed successfully

[jertel]

I've already fixed that error and so the next release will avoid crashing the rule in that situation. You can try it now by using the nightly build. To do so, switch to the elastalert2:latest Docker image tag and be sure to run docker pull first if you were already using latest.

[lufimoreira94]

That solved it, thank you! I just don't understand why it doesn't send any logs. After the pull it sent 14 requests, but now it went silent again. Any clue? I find it hard to believe that there are 0 matches to a spike of 100% for both up and down...

elastalert2-1 | 2025-01-23 12:37:41,412 INFO Running job "Rule: Under Performance Error Alert v2 (trigger: interval[0:01:00], next run at: 2025-01-23 12:38:45 UTC)" (scheduled at 2025-01-23 12:37:41.411913+00:00)
elastalert2-1 | 2025-01-23 12:37:41,428 DEBUG http://172.17.0.1:9200 "POST /.ds-traces-apm-default-*/_search?ignore_unavailable=true&size=0 HTTP/1.1" 200 None
elastalert2-1 | 2025-01-23 12:37:41,448 DEBUG http://172.17.0.1:9200 "POST /elastalert_status_status/_doc HTTP/1.1" 201 180
elastalert2-1 | 2025-01-23 12:37:41,449 INFO Ran Under Performance Error Alert v2 from 2025-01-23 12:36 UTC to 2025-01-23 12:37 UTC: 271 query hits (0 already seen), 0 matches, 0 alerts sent
elastalert2-1 | 2025-01-23 12:37:41,449 INFO Under Performance Error Alert v2 range 60
elastalert2-1 | 2025-01-23 12:37:41,449 INFO Job "Rule: Under Performance Error Alert v2 (trigger: interval[0:01:00], next run at: 2025-01-23 12:38:45 UTC)" executed successfully

[lufimoreira94]

I don't want to be a nuisance, but it still throws an error, but on the next line:

elastalert2-1 | File "/usr/local/lib/python3.13/site-packages/elastalert/ruletypes.py", line 528, in find_matches
elastalert2-1 | ref < self.rules.get('threshold_ref', 0)):
elastalert2-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
elastalert2-1 | TypeError: '<' not supported between instances of 'NoneType' and 'int'

I suppose "ref is not None" on the if would solve the problem.

[jertel]

That's a good find. This is a scenario where you have sufficient consecutive time windows that have no hits, and so not only is the current time window empty, but so is the reference time window. Thanks for posting this and I'll work on another PR tonight. If you'd like to submit one before then, feel free. You can use my recent PR as an example.

[lufimoreira94]

Done. Thanks for all your help, you're a legend.

[lufimoreira94]

I'm sorry for coming back to this ticket, but I have a question. Does Spike Aggregation compare the average of the current values in the timeframe to the average of the values in the buffer_time? Or all the values of said timeframe individually? If the former, how would I compare it individually?

[jertel]

It compares the mean of the current timeframe to the mean of the reference timeframe.

[lufimoreira94]

How would I compare the individual values? Imagine, I would like to know if said trace took more time than it should, comparing it to the average. If I compare the mean to the mean, if, in the timeframe, I have a value that is too low, it might ignore the high value.

[jertel]

Perhaps use metric_agg_type: max?