Enhance datadog search with other APM capabilities

[robrap]

Here is the datadog search script:

• https://github.com/edx/edx-arch-experiments/blob/main/edx_arch_experiments/datadog_monitoring/scripts/datadog_search.py

Acceptance Criteria:

• Document or implement missing areas.

Known missing areas:

• APM generated metrics: https://app.datadoghq.com/apm/traces/generate-metrics
• APM Retention filters: https://app.datadoghq.com/apm/traces/retention-filters

Implementation Notes:

• In general, the script should make it relatively simple to add a new feature by just adding the results of a different DD API call, assuming there is an appropriate API call.
• It is possible that only an Admin user can run some of these APIs? If so, we could document the limitations, and use an Admin to test the changes (if not implement). Please research first and document.
• If we decide that sustained alerts requires more generated metrics, this may become a priority.
• If too much work is involved to implement, we may want to update the how-to and/or script to explain the limitation (e.g. known missing areas that should be manually reviewed).