Numaflow Debuggability

[veds-g]

Summary

Currently, the numaflow UI offers real-time status updates at various levels, including vertex, pod, and container levels with enough context. To enhance debuggability, we should extend this capability to include more comprehensive and detailed metrics with historical data. By analyzing the metrics over time, users can detect trends, identify anomalies, and determine the root cause of issues.

Tasks

• unify metrics to avoid duplicate metrics #2218
• test querying our existing metrics with a metrics server supporting PromQL
• Backend: define an API to run PromQL query from the Numaflow Server #2058
• finalise a charting library to render graphs
• UI: Show Sink Invocation Latency for a Monovertex #2104
• UI: Allow users to filter the time series by time #2107
• UI: Allow users to add pod as a dimension(group by pod) #2105
• UI: Allow users to filter the time series by pod #2106
• Enhance UI to visualise counter metrics #2242
• Enhance UI to visualise gauge metrics #2243

[vigith]

Why do we need a prometheus server in Numaflow? Can't we assume/mandate that a metrics provider that supports PromQL is provided?

[veds-g]

Why do we need a prometheus server in Numaflow? Can't we assume/mandate that a metrics provider that supports PromQL is provided?

We do not need this. Supported metrics provider will be a mandate. This issue is just about testing promql query by running a prometheus server. Maybe we can rename it?