ListEventGroup API response is slow, when no CEL filter or keyword search results are found

[nitamanna18]

Describe the bug
When the ListEventGroup API cannot find results, the response time increases significantly, taking up to 54 seconds. However, when results are found, the API responds almost instantly (within milliseconds).
We made only one call from Origin Reports Adaptor (screenshot below) but we noticed 3 request to Kingdom log(screenshot below) for the same single request which returns the same result within 600 ms.

Kingdom Log:

Origin Reports Adaptor Log

Steps to reproduce

1. Login to Origin Platform
2. Go to Campaign group page
3. Search for Event Groups without any filters and the EGs not found in UI
4. Click on “Search“ button to call listeventgroup API

Component(s) affected
Kingdom, Reporting Server

Version
Halo 5.11

Environment
Origin Stage & PROD Environment

Additional context
attached log screenshot above.

[SanjayVas]

We made some performance enhancements to ListEventGroups in v0.5.13 as part of fixing a previous simulator issue. See #1923. Feel free to reopen if you still see the same problem after upgrading past that version.

The behavior is that it attempts to read as many results as it can before the RPC deadline, assuming that the overhead for calling the Kingdom is 100ms¹. If the caller has not specified a deadline², the service will use 30s.

As an aside, all issues filed here should be in the context of Halo software only. This means that the "Step to reproduce" should only cover parts of Halo. For the purposes of filing issues here, assume that the Origin UI doesn't exist.

Footnotes

1. We should probably go back and make this configurable ↩

2. This should ideally never be true, as the caller should always specify an RPC deadline ↩