NATS JetStream Sharding

[c1moore]

I don't know if this is a suggestion or a discussion topic, but I've been trying to determine ways to shard streams across a cluster to spread out load across the cluster.

According to the community, NATS itself does not try to spread streams across different instances. It basically chooses the instances that respond the fastest, which may or may not implicitly spread streams out. It seems to me that for this use case (sharding), that wouldn't happen since chances are you're creating all the streams around the same time and there probably aren't any consumers yet.

So the next suggestion is to leverage tags. What I'd like to end up with is something like this

However, I haven't found a way to consistently generate tags in an automated fashion in Kubernetes, especially using the official NATS helmchart, that doesn't require an external service/state just to coordinate the instances (e.g. "you are the second instance in AZ 1, you will be assigned group 2"). This becomes much more complicated if you start considering the potential for instances to go down and multi-AZ replication. In the end, I'd rather not maintain something external just to tag NATS servers.

So, what's the recommended strategy for having a setup like this? Alternatively, the problem goes away if NATS could direct streams to instances that have the fewest streams already by default (and let this be overridden by tags).

[jnmoyne]

About stream sharding and their distribution over the cluster: the selection of the peers at stream creation time is load balanced (it looks at the current distribution of JS assets when making the selection). Of course this is only as longs a the number of nodes (matching whatever tags you may have) is higher than the replication degree for the stream.

About assigning placement tags using a helmchart is a bit complicated but possible by using initContainers to create a set of tags in a server-tags/tags.conf file. The example below uses kubectl to generate the tags.

config:
  cluster:
    enabled: true
    replicas: 3
    
  merge:
    $include: server-tags/tags.conf

container:
  patch:
  - op: add
    path: /volumeMounts/-
    value:
      name: server-tags
      mountPath: /etc/nats-config/server-tags

podTemplate:
  merge:
    spec:
      initContainers:
      - name: server-tags
        image:
          $tplYaml: >-
            bitnami/kubectl:{{ .Capabilities.KubeVersion.Major }}.{{ .Capabilities.KubeVersion.Minor }}
        command: ["sh", "-ec"]
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        args:
        - |-
          kubectl get node "$NODE_NAME" --output json \
            | jq '{"server_tags": [ .metadata.labels | to_entries | .[] | .key + ":" + .value ]}' \
            > /server-tags/tags.conf
          echo "wrote server tags"
        volumeMounts:
        - name: server-tags
          mountPath: /server-tags
  patch:
  - op: add
    path: /spec/volumes/-
    value:
      name: server-tags
      emptyDir: {}

extraResources:
- apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRole
  metadata:
    name:
      $tplYaml: >-
        {{ include "nats.namespace" $ }}-{{ include "nats.fullname" $ }}-get-node
  rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get"]
- apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRoleBinding
  metadata:
    name:
      $tplYaml: >-
        {{ include "nats.namespace" $ }}-{{ include "nats.fullname" $ }}-get-node-binding
  subjects:
  - kind: ServiceAccount
# fix - the nats helm chart uses the 'default' ServiceAccount instead
#    name:
#      $tplYaml: >-
#        {{ .Values.serviceAccount.name }}
    name: default
    namespace:
      $tplYaml: >-
        {{ include "nats.namespace" $ }}
  roleRef:
    kind: ClusterRole
    name:
      $tplYaml: >-
        {{ include "nats.namespace" $ }}-{{ include "nats.fullname" $ }}-get-node
    apiGroup: rbac.authorization.k8s.io