ws usage inside kubernetes

[jeluard]

I'm trying to use Pod.exec (so using the ws feature) inside a kubernetes cluster and get the following error:

failed to upgrade to a WebSocket connection: failed to switch protocol

The call itself is done and consumed within a Pod.

Any clue what could explain such behaviour?

[nightkr]

The error message implies that the traffic is going through a (reverse?) HTTP proxy that doesn't support websockets.

What K8s version are you running? What kind of cluster is it (managed, kubespray, openshift, "the hard way", etc...)? Is internet traffic tunneled through a proxy? Are you using in-cluster configuration?

[jeluard]

My understanding of kubernetes internal networking is a bit limited, unsure how a proxy would be involved in my case.

I'm using the in-cluster config. The internet traffic is handled via ingress-nginx (conf here)

Using Google Kubernetes engine v1.21.10-gke.2000.

[jeluard]

Coming back to this: in my use case one pod calls Pod.exec of another pod from the same GKE cluster, but running in a different nodepool (so different nodes).

Any hint on how I could debug this?

[Venryx]

Just wanted to mention that I hit the same error you hit, when trying to use Pod.exec.

I eventually found a solution: The main thing being that rather than calling the provided exec function directly, you construct the request manually, and then just use the client.connect(req) function that kube provides to handle the websocket upgrade process:

pub async fn exec_command_in_another_pod(pod_namespace: &str, pod_name: &str, container: Option<&str>, command_name: &str, command_args: Vec<String>, allow_utf8_lossy: bool) -> Result<String, Error> {
    info!("Beginning request to run command in another pod. @target_pod:{} @command_name:{} @command_args:{:?}", pod_name, command_name, command_args);
    let token = fs::read_to_string("/var/run/secrets/kubernetes.io/serviceaccount/token")?;

    let mut query_str = format!("?command={}", command_name);
    for arg in &command_args {
        query_str.push_str(&format!("&command={}", arg));
    }
    if let Some(container) = container {
        query_str.push_str(&format!("&container={}", container));
    }
    query_str.push_str("&stdin=true&stderr=true&stdout=true&tty=true");

    let req = tungstenite::http::Request::builder().uri(format!("https://kubernetes.default.svc.cluster.local/api/v1/namespaces/{}/pods/{}/exec{}", pod_namespace, pod_name, query_str))
        .method("GET")
        .header("Authorization", format!("Bearer {token}"))
        .body(vec![]).unwrap();

    // this constructor automatically finds the k8s auth-data from environment (eg. token from "/var/run/secrets/kubernetes.io/serviceaccount/token", and k8s host/port from env-vars and/or default service uri)
    let client = Client::try_default().await?;

    let mut response = client.connect(req).await?;
    let mut res_as_str = String::new();
    loop {
        let (next_item, rest_of_response) = response.into_future().await;
        response = rest_of_response;
        match next_item {
            Some(Ok(item)) => {
                let item_into_text = item.into_text()?;
                let item_as_str = item_into_text.as_str();
                res_as_str.push_str(&item_as_str);
            }
            Some(Err(e)) => return Err(e.into()),
            None => break,
        }
    }

    info!("Got response from k8s server, on trying to run command using exec. @command:\"{} {}\" @response_len: {}", command_name, command_args.join(" "), res_as_str.len());
    Ok(res_as_str)
}

I don't know why the provided exec function doesn't work, but the workaround above gets the job done without too much boilerplate.

Anyway, you can find the full code here: https://github.com/debate-map/app/blob/60dd0a63563bbf57e7d863ef7429ab751b22cbe2/Packages/rust-shared/src/utils/_k8s.rs#L121

And an example usage of the function can be seen here: https://github.com/debate-map/app/blob/60dd0a63563bbf57e7d863ef7429ab751b22cbe2/Packages/app-server/src/db/general/backups.rs#L54

Hope this helps.

[jeluard]

Cool, looks great!

[vCassius]

i have same question. it's not work inside in k8s cluster.it's strange.but if run outside it's working!