[Configuration] Environment variable substitution opt-in via OTEL_CONFIG_INTERPOLATE

[pellared]

What are you trying to achieve?

For SDKs, make the environment variable substitution (also known as interpolation) an opt-in feature controlled via a OTEL_CONFIG_INTERPOLATE boolean env var.

Additional context.

The env var substitution logic is complex.

Requiring each SDK to [correctly] implement it may take a long time. Personally, I also think that because of the complexity of the problem each language should have fuzz tests for the parsing logic.

Making a different substitution logic in SDKs and Collector may result in a bad user experience. Getting different results for the same config in different OTel components can cause a lot of user frustration.

Users can use envsubst or a tool provided by OTel like validator (probably would be good to rename it) instead of doing it via the SDK.

The environment variable substitution as part of the SDK is a blocker only for environments where the ecosystems cannot bundle in a binary specific to a particular platform / architecture. However, the same can be said for the Collector and some missing functionalities in the SDKs.

Making environment variable substitution as an opt-in feature should speedup the configuration development that should satisfy most of the users without shutting the door for future improvements.

Related PR where this idea has been originally proposed: #4375

[jack-berg]

For SDKs, make the environment variable substitution (also known as interpolation) an opt-in feature controlled via a OTEL_CONFIG_INTERPOLATE boolean env var.

Users can use envsubst or a tool provided by OTel like validator (probably would be good to rename it) instead of doing it via the SDK.

Just to restate my understanding of the motivation here (correct me if I'm wrong):

• Don't make env var sub mandatory for SDKs, since users can instead use an initialization script with the validator to perform substitution
• If a user does this, and the SDK later adds env var substitution, then performing substitution a second time may produce unintended results
• Therefore, the user sets OTEL_CONFIG_INTERPOLATE=false, indicating that they prefer to do it themselves

I think this makes sense in theory, but in practice, compared to the overall complexity of implementing declarative config, the complexity of env var substitution is small. Additionally, because the logic has very clear expected inputs and outputs, its easy to test, and gain confidence in the implementation. I.e. maintainers can reference test suites from existing implementations to bootstrap their own test suites.

Personally, I haven't heard feedback about env var substitution complexity from any of the 5 language implementations.

Suppose a language implementation does find env substitution complex AND is unable to bundle the OTel validator tool. For OTEL_CONFIG_INTERPOLATE to be useful, they would also need to be in a hurry to deliver a stable declarative config implementation, since it would be perfectly valid for the language to make an experimental implementation available without support for env var substitution. I find it hard to imagine a language implementation being in such a rush to stabilize that they can't spend some small amount of extra time on an env var substitution implementation.

@open-telemetry/configuration-approvers WDYT?

[pellared]

We discussed it during OTel Go SIG and in summary is that if we add something like OTEL_CONFIG_INTERPOLATE would steepen the learning curve for OTel users and it is better to add env var substitution everywhere instead of giving different behavior depending on some yet another config.

If we find the env var subsitution logic unnecessarily complex then we should rather simplify it for all OTel components (including OTel Collector).

Closing the issue as I am no longer in favor of my proposal.