air-gapped environment Runtime environment dependencies: local Https PKI and BasicAuth

[shalberd]

Is your feature request related to a problem? Please describe.
Similar to issues related to Airflow components and package components, where requests.get was used without a possibility to add Http Basic Auth initially and without possibility to add a private PKI CA Certificate as trusted.

In the documentation to runtime environment dependencies https://elyra.readthedocs.io/en/latest/recipes/running-elyra-in-air-gapped-environment.htmt, it is mentioned that during pipeline execution in the Kubeflow Pipelines or Apache Airflow environment, a different location for e.g. ELYRA_BOOTSTRAP_SCRIPT_URL, ELYRA_REQUIREMENTS_URL_PY37 (after Elyra 3.6) and ELYRA_REQUIREMENTS_URL can be added as an env file.

A hard requirement so far is ""you must store a copy of these files in a location that is accessible via an anonymous HTTP GET request and configure the following environment
variables in the environment where JupyterLab is running:""

This can be seen in

https://github.com/elyra-ai/elyra/blob/main/elyra/airflow/operator.py#L98 where the values of the ENV files are referenced and then downloaded into the filesystem with curl for airflow and

for kubeflow pipeline runtime dependencies in

https://github.com/elyra-ai/elyra/blob/main/elyra/kfp/operator.py#L194

That is not practical in an enterprise environment, similar to the PKI non-trusted issue.

The curl commands implemented will not work with Http Basic Auth and also not with https locations having a non-publicly trusted CA and/or intermediate CA.

Describe the solution you'd like
A solution based on curl, or even better yet requests.get, that supports Http Basic Auth with user id and password or api key.
A solution that also supports trusting custom CA certificates mapped in the container filesystem.

Describe alternatives you've considered
Authentication is a must, and internal PKI is also very commin for SSL certificates of systems in enterprise environments.
I see no alternative.

Additional context
Add any other context or screenshots about the feature request here.

[ptitzler]

Related: #2805 (investigate removal of remote file dependencies)

[shalberd]

solved with #2912 and #2797