[Entity Analytics] [Entity Store] [API] Changes to support event.ingested as a configurable timestamp files for init and enable endpoints

[abhishekbhatia1710]

Summary

This PR introduces support for configuring the timestamp field for entity store enablement.

By default, the timestamp field is set to @timestamp, but users can opt to use event.ingested or another preferred value based on their requirements.

Entity Store API changes

Entity Store enable

POST kbn:/api/entity_store/enable
{
  "timestampField": "event.ingested"
}

Result

{
  "engines": [
    {
      "status": "started",
      "type": "user",
      "indexPattern": "",
      "filter": "",
      "fieldHistoryLength": 10,
      "timestampField": "event.ingested"
    },
    {
      "status": "started",
      "type": "host",
      "indexPattern": "",
      "filter": "",
      "fieldHistoryLength": 10,
      "timestampField": "event.ingested"
    },
    {
      "status": "started",
      "type": "universal",
      "indexPattern": "",
      "filter": "",
      "fieldHistoryLength": 10,
      "timestampField": "event.ingested"
    }
  ],
  "status": "running"
}

POST kbn:/api/entity_store/enable
{
}

Result

{
  "engines": [
    {
      "status": "started",
      "type": "host",
      "indexPattern": "",
      "filter": "",
      "fieldHistoryLength": 10,
      "timestampField": "@timestamp"
    },
    {
      "status": "started",
      "type": "user",
      "indexPattern": "",
      "filter": "",
      "fieldHistoryLength": 10,
      "timestampField": "@timestamp"
    },
    {
      "status": "started",
      "type": "universal",
      "indexPattern": "",
      "filter": "",
      "fieldHistoryLength": 10,
      "timestampField": "@timestamp"
    }
  ],
  "status": "running"
}

Different entity types can have distinct timestampField values, as described below:

For instance, the host entity uses event.ingested as its timestampField, while other entities default to the @timestamp field.

POST kbn:/api/entity_store/engines/host/init
{
  "timestampField": "event.ingested"
}

POST kbn:/api/entity_store/engines/user/init
{
}

Result

{
  "engines": [
    {
      "status": "started",
      "type": "universal",
      "indexPattern": "",
      "filter": "",
      "fieldHistoryLength": 10,
      "timestampField": "@timestamp"
    },
    {
      "status": "started",
      "type": "host",
      "indexPattern": "",
      "filter": "",
      "fieldHistoryLength": 10,
      "timestampField": "event.ingested"
    },
    {
      "status": "started",
      "type": "user",
      "indexPattern": "",
      "filter": "",
      "fieldHistoryLength": 10,
      "timestampField": "@timestamp"
    }
  ],
  "status": "running"
}

Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

• This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The release_note:breaking label should be applied in these situations.
• Flaky Test Runner was used on any tests changed
• The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines

Testing steps

• Check out this PR branch.
• Start Kibana.
• Avoid enabling the Entity Store from the UI initially. Instead, enable it via the API by following the steps outlined below.

DELETE kbn:/api/entity_store/engines/user
DELETE kbn:/api/entity_store/engines/host

GET kbn:/api/entity_store/status

 All engines with the same timestamp Field
POST kbn:/api/entity_store/enable
{
  "timestampField": "event.ingested"
}

# All engines with the same timestamp Field
POST kbn:/api/entity_store/enable
{
}

# Different entity with different timestamp Field
POST kbn:/api/entity_store/engines/host/init
{
  "timestampField": "event.ingested"
}

POST kbn:/api/entity_store/engines/user/init
{
}

4. Afterward, you can test the functionality through the UI. First, clear the entity data from the UI, then start the Entity Store. Note that the UI currently does not offer a way to configure this parameter.

[elasticmachine]

Pinging @elastic/security-entity-analytics (Team:Entity Analytics)

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: 4b118da

Failed CI Steps

• Checks
• Check Types

History

• 💔 Build #270038 failed 3727e92
• 💔 Build #270030 failed 134d938

cc @abhishekbhatia1710