How to assign nested match value "system.load.norm.5" or "{18}" to opsgenie_message?

[larsl-nf]

Q: Why can't I use "system.load.norm.5" or "{18}" in "opsgenie_message" assignment? (it works fine in "alert_text")
The value is visible in match json, using "print (f"{match}")" in enhancement.
I get these errors when trying to access the value, see test notes below.

• "KeyError: 5"
• 'dict' object has no attribute 'load'

--- MATCH JSON ---
{
   "target_index":"metricbeat-7.17.6",
   "env":"production",
   "system":{
      "load":{
         "1":1.04,
         "cores":1,
         "15":0.8,
         "5":1.01,
         "norm":{
            "1":1.04,
            "15":0.8, 
            "5":1.01       <-- I WANT TO GET THIS VALUE: system.load.norm.5 = "1.01"
         }
      }
   },
<-- CUT -->
}

--- RULE.YAML ---
name: "CPU Saturation High"
# File: mb_cpu_saturation_high.yaml

index: "metricbeat-7*"
use_strfttime_index: true

type: "frequency"
query_key: "agent.hostname"
num_events: 15      
timeframe:
  minutes: 30
filter:
  - range:
      "system.load.norm.5":
        gte: 0.85              # Default: 0.85, Testing rule: 80

match_enhancements:
  - "elastalert_modules.ops_enhancements.MyEnhancement"

alert_text_args:           #      
  - "networkName"          #  {0} 
  - "networkId"            #  {1}
  <- CUT -->
  - "system.load.norm.5"   # {18}          <-- ASSIGNED HERE to {18}
  - "system.load.5"        # {19}   

alert_text_type: "alert_text_only"
alert_text:
  <tr><td> networkName </td><td> {0} </td></tr>
  <tr><td> networkId </td><td> {1} </td></tr>
  <--- CUT --->
  <tr><td> system.load.norm.5 </td><td> {18} </td></tr>       <-- Prints out proper value in Opsgenie alert
  <tr><td> system.load.5 </td><td> {19}</td></tr>

realert:
  hours: 6
alert:
  - "opsgenie"
opsgenie_key: "xxx"
opsgenie_subject: "CPU saturation high"
opsgenie_message: 'CPU saturation high (>85%) - {system[load][norm][5]} - {networkName}'  <-- Trying to use value here.

TESTING DONE:

THIS WORKS: (but I need to drill down one more level)

- opsgenie_message: 'CPU saturation high (>85%) - {system[load][norm]} - {networkName}'
  OK:  CPU saturation high (>85%) - {'1': 0.895, '15': 0.795, '5': 0.91} - MyNetwork01     <-- Nearly there, I want the value "0.91"

TESTED WITH THESES VARIATIONS:

- opsgenie_message: 'CPU saturation high (>85%) - {system[load][5]} - {networkName}' -> ERROR: KeyError: 5
- system[load][norm]["5"]}-> ERROR: KeyError: '"5"'
- {system[load][norm][5]} -> ERROR: "KeyError: 5"
- {18}                    -> ERROR: IndexError: Replacement index 18 out of range for positional args tuple
- {system.load.norm.5}    -> ERROR: 'AttributeError: 'dict' object has no attribute 'load'  
- {system.load}           -> ERROR: AttributeError: 'dict' object has no attribute 'load'
- {system.load.norm}      -> ERROR: AttributeError: 'dict' object has no attribute 'load'
- {system.load.norm[0]5}  -> ERROR: AttributeError: 'dict' object has no attribute 'load'
- {system.load.norm.5}    -> ERROR: AttributeError: 'dict' object has no attribute 'load'
- {system.load.5}         -> ERROR: AttributeError: 'dict' object has no attribute 'load' 

[jertel]

It works in alert_text because ElastAlert 2 is coded to perform an advanced lookup algorithm and variable/arg substitution using the given arg list (or Jinja templating if configured).

It does not work in opsgenie_message because the author of that alerter chose to use the standard Python str.format() syntax for var substitution, which appears to be incapable of handling numeric dict keys.

To achieve your desired opsgenie message, you should omit the opsgenie_message override from your rule config. This will allow opsgenie alerter to fallback to the default algorithm, using the alert_subject and alert_subject_args instead.

To do this, remove the opsgenie_message parameter from your rule config, and add the following:

alert_subject: 'CPU saturation high (>85%) - {0} - {1}'
alert_subject_args:
  - "networkName"  # {0}
  - "system.load.norm.5"   # {1}

[larsl-nf]

Thank you Jertel for the quick response, now I know what the issue is. Please mark this issue as resolved.
I tried different combinations of your solution but could not get the "title" (opsgenie_message:) of the alert to change. I will instead use the enhancement script to assign "system.load.norm.5" = "mysysload" and use "mysysload" in my rule.

# Sample code change I tried, tried different combinations and locations of new code block
alert_subject_args:
  - "networkName"  # {0}
  - "system.load.norm.5"   # {1}
alert_subject: 'New CPU saturation high Value: {1}) - {0}'

# Alerter Configuration #
realert:
  hours: 6
alert:
  - "opsgenie"
opsgenie_key: "zzz"
opsgenie_subject: "CPU saturation high"                          # Tried also commenting out opsgenie_subject
# opsgenie_message: 'CPU saturation high (>85%) - {system[load][norm]["5"]} - {networkName}' <-- COMMENTED OUT
# I also tried alert_subject_args and alert_subject code block here.
opsgenie_alias: "CPU saturation high (>85%) - {networkName}"
opsgenie_priority: "P2"
opsgenie_teams:
  - "Technical Support"

[jertel]

Sorry, I should have written this:

opsgenie_subject: 'CPU saturation high (>85%) - {0} - {1}'
opsgenie_subject_args:
  - "networkName"  # {0}
  - "system.load.norm.5"   # {1}

[larsl-nf]

Yihaaa, works great! Thank you, will be very useful for our alerts. Here is my final config:

# ##################### #
# Alerter Configuration #
# ##################### #

realert:
  hours: 6
alert:
  - "opsgenie"
opsgenie_key: "xxx"
opsgenie_subject: "CPU saturation high (>85%, Value: {1}) - {0}"
opsgenie_subject_args:
  - "networkName"  # {0}
  - "system.load.norm.5"   # {1}
# opsgenie_message: 'CPU saturation high (>85%) - {networkName}'  # REMOVE / COMMENT OUT
opsgenie_alias: "CPU saturation high (>85%) - {networkName}"
opsgenie_priority: "P2"
opsgenie_teams:
  - "Technical Support"
opsgenie_details:
  networkName: { field: networkName }
  networkId: { field: networkId }
  networkGroupId: { field: networkGroupId }
  organizationId: { field: organizationId }