Use LD_PRELOAD for setting environment variables e.g. JAVA_TOOL_OPTIONS / OTEL_

[pavolloffay]

Component(s)

No response

Is your feature request related to a problem? Please describe.

At the moment the operator sets the env variables directly on the pod spec. The downside is if user defines the same variables in docker image or in config map. The operator does not know anything about those env vars. The solution might be to use LD_PRELOAD to set and override env vars.

The same approach is used by Lumigo operator and probably other vendors. https://github.com/lumigo-io/lumigo-kubernetes-operator/blob/88a3f78e5a32b965210659db11ef97f2a0c6f6c6/controller/src/mutation/mutate.go#L50

This would also resolve #1814

Describe the solution you'd like

Use LD_PRELOAD to set env vars for the instrumentation CR.

Describe alternatives you've considered

No response

Additional context

No response

[yuriolisa]

@pavolloffay, are you working on it or can I take this one?

[pavolloffay]

I haven't started. I think this needs maybe a POC and see what are the other implications.

[mmanciop]

OG of the Lumigo operator here :-)

We do this in the Dash0 operator (fully OSS), and while I personally built it as well in the Lumigo operator a couple of years ago, the injector there is not OSS iirc. We have used LD_PRELOAD technology at scale at Instana as well, and Dynatrace has been doing the same all along with their OneAgent. The implications mostly are three in my experience:

• The LD_PRELOAD object must be statically linked or you break anything that uses a different flavour of libc (yes, musl libc, I am looking at you; but older GLIBC versions are finicky as well IIRC)
• Unsupported Python versions, especially v2.7 and early v3, but they can be overcome with site-specific hooks
• Conflicting instrumentations built-in in the applications, but this is an issue that the current OpenTelemetry operator already faces

(Additional tidbits in this blog post.)

From Dash0's side, we'd be delighted to see LD_PRELOAD mechanics being adopted in the OpenTelemetry operator, and would love to help this happen.

[mmanciop]

Besides, the LD_PRELOAD technology is the same regardless of whether the injected runtime lives in a container or is running directly on the Linux host. If we develop this tech in the community, we would be able to provide, for example, DEB or RPM packages with all types of OTel automatic instrumentations that just work!

[pavolloffay]

@mmanciop thanks for looping in. We would appreciate collaboration on this feature 👍🏼

https://lumigo.io/blog/the-magic-behind-the-lumigo-kubernetes-operator/ mentions that the init-container is also used to copy the instrumentation/SDK. Does it mean that the docker image for the init container needs to bundle instrumentation for all languages?

[mmanciop]

@mmanciop thanks for looping in. We would appreciate collaboration on this feature 👍🏼

https://lumigo.io/blog/the-magic-behind-the-lumigo-kubernetes-operator/ mentions that the init-container is also used to copy the instrumentation/SDK. Does it mean that the docker image for the init container needs to bundle instrumentation for all languages?

Yes, the init container image must bundle all languages that the LD_PRELOAD can activate. And of course the LD_PRELOAD object itself.

[berezinsn]

Hi there,

I would like to inquire about any updates or progress on the feature related to concatenating JAVA_TOOL_OPTIONS with OpenTelemetry auto-instrumentation agent options.

As of now, it seems that it is only possible to concatenate environment variables explicitly defined within the deployment YAML under the env section. However, it appears that concatenation of the values from JAVA_TOOL_OPTIONS with the OpenTelemetry agent options (e.g., -javaagent or other configuration flags) is still not supported or automatic.

Could you please clarify whether this functionality is in the works or if there are any workarounds available? Any update would be greatly appreciated!

Thank you for your time and efforts.

[mmanciop]

Hi @berezinsn , it's a functionality we have working with the Dash0 operator and that I would love to contribute upstream. In your specific case, you maybe also use the _JAVA_OPTIONS environment variable, which is used by JVMs pretty much the same way JAVA_TOOL_OPTIONS is. Hope this helps in the meanwhile.