Skip to content

Latest commit

 

History

History
127 lines (87 loc) · 5.72 KB

20240517-R-http-metrics-path-matching.md

File metadata and controls

127 lines (87 loc) · 5.72 KB

HTTP Metrics Path Matching

  • Author(s): @nelson-parente @jmprusi
  • State: Ready for Implementation
  • Updated: 2024-05-17

Overview

This is a design proposal to implement a new opt-in API for path matching within Dapr HTTP metrics. By enabling path matching users can define paths that will be matched and replaced without being at risk of unbounded path cardinality and other issues that motivate the introduction of the low cardinality mode in Dapr. This will enable users to have more meaningful and manageable metrics in a controlled way.

Background

In #6723, Dapr reduced the cardinality of its HTTP metrics in order to address memory issues users reported and restrain unbounded path cardinality which posed as a security threat. This change introduced two cardinality modes (high/low) controlled by the increasedCardinality flag.

The caveat with low cardinality is that it dropped paths since they were one of the sources for the high cardinality. While this is a reasonable approach, it leads in the loss of important data needed for monitoring, performance analysis, and troubleshooting. To address this, we opened #7719.

This proposal introduces an opt-in API that allows users to define the paths that matter the most, effectively adding matched paths to metrics without relying on regex's, which are known to be CPU-intensive.

With this API, users will be able to configure path matching through a simple interface, providing the paths they care about and tailoring metrics to their specific requirements without compromising memory and security issues.

Related Items

Related issues

Initial low cardinality issue: #6723 Issue related with low cardinality dropped metrics data: #7719

Expectations and alternatives

The proposed solution adds value to users' observability without compromising security and memory usage. The API is designed to be simple to configure, allowing users to configure the paths they care about. We considered other regex-based solutions but these are known to be CPU-intensive and can lead to performance degradation.

Implementation Details

Solution

This proposal introduces an opt-in API for path matching within Dapr HTTP metrics. The goal is to offer a way to match and include paths in the metrics without relying on CPU-intensive regex's and with a guarantee that path cardinality is controlled.

spec:
  metric:
    enabled: true
    http:
      increasedCardinality: true
      pathMatching:
        - /orders/{orderID}/items/{itemID}
        - /users/{userID}
        - /categories/{categoryID}/subcategories/{subCategoryID}
        - /customers/{customerID}/orders/{orderID}
Examples

Examples of how the Path Matching API can be used in the metrics. The examples compare the metric dapr_http_server_request_count with the possible configuration combinations: low and high cardinality, with and without path matching.

  • Low Cardinality Without Path Matching
http:
  increasedCardinality: false
dapr_http_server_request_count{app_id="ping",method="InvokeService/ping",status="200"} 5
  • Low Cardinality With Path Matching
http:
  increasedCardinality: false
  pathMatching:
    - /orders/{orderID}
dapr_http_server_request_count{app_id="ping",method="GET",path="/orders/{orderID}",status="200"} 4
dapr_http_server_request_count{app_id="ping",method="GET",path="",status="200"} 1
  • High Cardinality Without Path Matching
http:
  increasedCardinality: true
dapr_http_server_request_count{app_id="ping",method="GET",path="/items/123456",status="200"} 1
dapr_http_server_request_count{app_id="ping",method="GET",path="/orders/1234",status="200"} 1
dapr_http_server_request_count{app_id="ping",method="GET",path="/orders/12345",status="200"} 1
dapr_http_server_request_count{app_id="ping",method="GET",path="/orders/123456",status="200"} 1
dapr_http_server_request_count{app_id="ping",method="GET",path="/orders/1234567",status="200"} 1
  • High Cardinality With Path Matching
http:
  increasedCardinality: true
  pathMatching:
    - /orders/{orderID}
dapr_http_server_request_count{app_id="ping",method="GET",path="/items/123456",status="200"} 1
dapr_http_server_request_count{app_id="ping",method="GET",path="/orders/{orderID}",status="200"} 4

Features

  • pathMatching where users can specify paths for path matching.

The path matching will use the same patterns as the Go standard library (see https://pkg.go.dev/net/http#hdr-Patterns), ensuring reliable and well-supported path matching.

When the increasedCardinality flag is set to false (default in 1.14), non-matched paths are transformed into a catch-all bucket to control and limit cardinality, preventing unbounded growth. On the other hand, when increasedCardinality is true, non-matched paths are passed through as they normally would be, allowing for potentially higher cardinality but preserving the original path data. This is the main difference in the feature behavior when used with low versus high cardinality.

This Path Matching API empowers users that rely on the metrics and observability scrapped from low cardinality that will soon be the default, providing a controlled means to manage path cardinality. Those indifferent may opt for low cardinality, while legacy high cardinality mode remains available for alternative needs.

Acceptance Criteria

  • The Path Matching API is successfully integrated into the Dapr runtime, allowing users to enable/disable path matching via configuration.

Completion Checklist

  • Implementation in daprd
  • API documentation
  • Integration, E2E tests