Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Metrics can't be send due to gRPC error ResourceExhausted using --out experimental-opentelemetry #4277

Closed
FilipeGoncalvesBasecone opened this issue Jan 24, 2025 · 1 comment
Assignees

Comments

@FilipeGoncalvesBasecone
Copy link

FilipeGoncalvesBasecone commented Jan 24, 2025

Brief summary

Using OpenTelemetry output and the option grpc it seems after a while it stops sending the metrics in our case to DataDog due to ResourceBeing Exhausted and it tried to send metrics file which has a size larger than the max allowed:

time="2025-01-24T10:54:27Z" level=info msg="2025/01/24 10:54:27 failed to upload metrics: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (44911808 vs. 4194304)"

The alternative for now is to use http exporter type instead.

Also worth to mention this bug was reported on xk6-output-opentelemetry extension:

k6 version

v0.56.0

OS

Windows 11, Ubuntu 22

Docker version and image (if applicable)

No response

Steps to reproduce the problem

Set up open telemetry output to use the default type: grpc
Execute the test with --out experimental-opentelemetry
After a while it will show on the console that the Resource has been exhausted while trying to upload the metrics.

Expected behaviour

It should upload the metrics seamlessly from start to finish of the test.

Actual behaviour

After a while it stops sending metrics due to the error ResourceExhausted

@olegbespalov
Copy link
Contributor

Hi @FilipeGoncalvesBasecone

Thanks for reporting, the experimental-opentelemetry currently is literally the https://github.com/grafana/xk6-output-opentelemetry, so let's consider grafana/xk6-output-opentelemetry#23 as the primary issue for the bug and continue there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants