Reduce resource consumption of discovery components

[mattdurham]

Reduce resource consumption of discovery components

Address high allocs in creation of targets lists

Targets/Discovery are represented as list(map[string]), with discovery running every 5 seconds in very large clusters the garbage collector can spike due to creating these labels. The names especially are repeated. The hard part is implementing the capsule value so hard coded targets still work. This is because in the alloy runtime there is a bug where you cannot convert from list/map to a capsule value.

In reviewing profiles this code was by far the largest allocator.

alloy/internal/component/discovery/discovery.go

Line 212 in 5845b2c

allTargets := []Target{}

Use case

Reduce memory and cpu consumption

[thampiotr]

@mattdurham I've re-purposed this issue for the work we've planned earlier this week.

[github-actions]

This issue has not had any activity in the past 30 days, so the needs-attention label has been added to it.
If the opened issue is a bug, check to see if a newer release fixed your issue. If it is no longer relevant, please feel free to close this issue.
The needs-attention label signals to maintainers that something has fallen through the cracks. No action is needed by you; your issue will be kept open and you do not have to respond to this comment. The label will be removed the next time this job runs if there is new activity.
Thank you for your contributions!

[thampiotr]

Some quick look at profiles to get an idea of the issue right now:

In one of our metrics collecting deployments we can see near 60% of CPU time spent on GC:

And near 60% of memory is allocated in discovery code:

This is a large cluster with 1k+ nodes and 20k+ targets.

[thampiotr]

Some low-hanging fruit fix that can be done with less effort for now: #2350

[thampiotr]

The above PR has reduced allocation in one cluster we have by 46% 🎉

The GC is down by 10-15%, but that cluster wasn't too stressed to begin with.

[thampiotr]

Another PR that will help reduce by 50% another allocation hotspot in Alloy: #2405
This time it's FanOut and Interceptor code. This can quickly grow in size in large pipelines.