How to run spark sql engine on kubernetes

[mikechengwei]

Hi Team. I want to know how to run spark sql engine on kubernetes, ref:docuemnt.

Question

Do I need to package the sql engine jar into a image and set it to conf spark.kubernetes.container.image?

[pan3793]

Not required.

The sql engine jar just play a role of examples.jar in the following example.

./bin/spark-submit \
    --master k8s://https://<k8s-apiserver-host>:<k8s-apiserver-port> \
    --deploy-mode cluster \
    --name spark-pi \
    --class org.apache.spark.examples.SparkPi \
    --conf spark.executor.instances=5 \
    --conf spark.kubernetes.container.image=<spark-image> \
    local:///path/to/examples.jar

[mikechengwei]

I still don't know how to run sql engine on kubernetes . I use sql engine jar instead of example jar , how should I fill in --class args?

[mikechengwei]

I think beeline tool connect to kyuubi, kyuubi will submit a long running application to kubernetes by spark-submit.

[pan3793]

yea.

[hddong]

I still don't know how to run sql engine on kubernetes . I use sql engine jar instead of example jar , how should I fill in --class args?

You can use kyuubi image, it contains both kyuubi spark engine jar and spark.

[zwangsheng]

If you want to run spark On k8s, you just need spark image and configuration your spark configs. https://spark.apache.org/docs/latest/running-on-kubernetes.html

[zwangsheng]

Kyuubi does not care you submit spark to which resource manager. spark.kubernetes.container.image just mean your Spark image.

[mikechengwei]

connect to kyuubi.

jdbc:hive2://tg-local-bdworker-1:10011/def> select timestamp '2018-11-17';
Unexpected end of file when reading from HS2 server. The root cause might be too many concurrent connections. Please ask the administrator to check the number of active connections, and adjusthive.server2.thrift.max.worker.threads if applicable.
Error: org.apache.thrift.transport.TTransportException (state=08S01,code=0)

engine log:

2022-03-23 19:25:57.975 INFO service.FrontendService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V10
2022-03-23 19:25:57.975 INFO session.SparkSQLSessionManager: Opening session for hadoop@10.2.1.6
2022-03-23 19:25:58.003 INFO session.SparkSQLSessionManager: hadoop's session with SessionHandle [4ae417bb-5f8d-4bb3-8fdf-ed512ba645bd] is opened, current opening sessions 1
2022-03-23 19:26:12.288 INFO service.FrontendService: Received request of closing SessionHandle [4ae417bb-5f8d-4bb3-8fdf-ed512ba645bd]
2022-03-23 19:26:12.288 INFO session.SparkSQLSessionManager: SessionHandle [4ae417bb-5f8d-4bb3-8fdf-ed512ba645bd] is closed, current opening sessions 0
2022-03-23 19:26:12.288 INFO service.FrontendService: Finished closing SessionHandle [4ae417bb-5f8d-4bb3-8fdf-ed512ba645bd]

executor status:

kubectl  get pods -n spark-operator
NAME                                                                           READY   STATUS      RESTARTS   AGE
kyuubiuserhadoop0b7ded6d-9d63-4f7b-85eb-a513b8d6c748-11ab2d7fb65e8b92-exec-1   1/1     Running     0          50m
kyuubiuserhadoop0b7ded6d-9d63-4f7b-85eb-a513b8d6c748-11ab2d7fb65e8b92-exec-2   1/1     Running     0          50m