DRAFT: new RBA Object - Step 3 - ESCU 5.0

[ljstella]

NOT YET READY FOR REVIEW

The tags.observable mechanism for RBA is unnecessary extra mental gymnastics, and is based on terminology that never materialized elsewhere. This PR will introduce a new top level field named rba that contains the RBA related fields and remove tags.observable, and tags.message.

Update: tags.confidence and tags.impact will stay in place and be documented as serving the purpose they defacto do today, which is the detection author's confidence in the analytic to find true positives, and the level of impact they have today, as well as being constructed as part of the calculation of notable severity. The validation confirming a risk score is the product of those two integers will not carry forward.

Example of this new object, using the bundled detection:

name: Anomalous usage of 7zip
id: 9364ee8e-a39a-11eb-8f1d-acde48001122
version: 1
...
rba:
  message: An instance of $parent_process_name$ spawning $process_name$ was identified
    on endpoint $dest$ by user $user$. This behavior is indicative of suspicious loading
    of 7zip.
  risk_objects: 
    - field: user
      type: user
      score: 56
    - field: dest
      type: system
      score: 56
  threat_objects:
    - field: parent_process_name
      type: parent_process_name
    - field: process_name
      type: process_name
tags:
...

Things to do:

• implement new RBA model
  • risk object type enum
  • port any existing risk object validation
  • threat object type enum
  • port any existing threat object validation
• remove all code and enums related to tags.observable OR SES
• Update packaged detection that gets generated with contentctl init
• integrate with in-flight testing work
• Determine the fate of tags.confidence and tags.impact
• Determine impacts to SSE API
• Determine migration mechanism (currently leaning towards none in-project and throwing validation errors, allowing folks to manually migrate and reconsider their RBA configs as they migrate- surely folks have pinned their dependencies, right?)
• Draft descriptions/docs for this new mechanism that can go in the release notes for when this gets a tagged release, so that folks can be prepared for the migration.
• Open a PR against splunk/security_content with changes to all existing content
• Update contentctl new wizard
• <insert next fun little surprise we left for later and need to fix here>

[pyth0n1c]

Right now, all of the RBA bits are required for every detection type. Do we want to require ALL fields for all types of detections? Or ONLY allow it (or make it optional) for certain types of detections?
In your example, I believe it's a coincidence that all risk/threat object fields appear in the message. This is not a requirement today and I don't think we want to start enforcing it, right?
I agree that, since this is a one-time migration, we don't need/want to build and maintain a tool to do that migration.

[ljstella]

@pyth0n1c - addressed in-line

Right now, all of the RBA bits are required for every detection type. Do we want to require ALL fields for all types of detections? Or ONLY allow it (or make it optional) for certain types of detections?

That's a good question. I want to see how easy it is to get the information we have from deployments at the time we are validating detections. It would be excellent in my POV if we could strip the RBA object out of analytics that won't have a risk config but require it on the ones that do. We'll see how easy that is to actually pull off. I'd like to do it that way because it allows us to change the default deployments, or for folks to layer in additional deployments, and have the "does this need RBA?" evaluation adapt as those changes are made, as opposed to a strict relationship on Detection Type.

In your example, I believe it's a coincidence that all risk/threat object fields appear in the message. This is not a requirement today and I don't think we want to start enforcing it, right?

That's correct- message fields don't have to be risk/threat objects (putting things like count into a message is a popular use, but would be silly as either a risk or threat). Likewise, you can add fields as threat or risk objects without directly referencing them in the message.

[ljstella]

New content testing branch: splunk/security_content#3269

Temporarily updated CI config to use that instead

[ljstella]

Changed the ESCU test branch back to rba_migration here: splunk/security_content#3204

We're about to merge the strict_yml content changes into the rba_migration branch, so we'll have just one branch to work against

[josehelps]

LGTM I trust Lou!