Kyuubi on K8s - Hive metastore information

[mmuru]

Kyuubi version - latest 1.5.2 and master custom build
Spark 3.2.1
Delta - 2.0.0
Kyuubi is shipped with Hive metastore 3.1.3

Please, some one provide the followings information

1. Hive metastore flow (from the client to server)
2. Kyuubi-service uses NodePort -works but how to access it externally? not via port-forward but DNS/LB
3. Kyuubi-service uses LoadBalancer - works and access it externally but throws the following error

2022-08-31 00:27:53.155 ERROR org.apache.thrift.server.TThreadPoolServer: Error occurred during processing of message.
java.lang.RuntimeException: org.apache.thrift.transport.TSaslTransportException: No data or no sasl data in the stream
	at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219) ~[libthrift-0.9.3.jar:0.9.3]
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269) ~[libthrift-0.9.3.jar:0.9.3]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_292]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_292]
	at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_292]

cc: @pan3793 @turboFei

[pan3793]

1. For no-kerberized deployment, only Spark Driver needs to talk w/ HMS, but HMS needs to talk w/the storage system(e.g. HDFS, S3) and DBMS(e.g. PG, MySQL).
2. It depends on. The network[1] is one of the most complex parts of K8s IMO. Basically, K8s requires a network CNI plugin to implement the K8s network model, and different implementations have different capabilities, the simplest way to access from external is to use LB, see answer 3.
3. LB is a totally vendor-dependents component, you should check the LoadBalancer document on how to reverse-proxy the Pods or maybe Services.

Any other context? Except for this error message, does it has other problems?

[1] https://kubernetes.io/docs/concepts/cluster-administration/networking/

[mmuru]

@pan3793: Thanks for your response.

1. using RDS PG as HMS and hive-site.xml in $SPARK_HOME/conf. Please, review attached hive-site.xml and let me know if any config missing.
   hive-site.xml.zip

2. ERROR message every 3 secs and slowing down the queries and sometime can't perform DDL. Kyuubi-svc uses AWS NLB and health check interval is 10secs.

Using Kyuubi frontend-protocols: THRIFT_BINARY,REST,MYSQL and listeners started fine. Please, let me know if you need any other information.

[mmuru]

3. Kyuubi-service type "Loadbalancer" needs the following annotations (aws-nlb)

    service.beta.kubernetes.io/aws-load-balancer-healthcheck-path: "/api/v1/ping"
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: "HTTP"
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: "10099"

after that ERROR org.apache.thrift.server.TThreadPoolServer: disappears.

[pan3793]

The error is expected if using HTTP client to access thrift endpoint