new docs: running KW at scale

[flavio]

It would be great to write some tips and tricks about how to run Kubewarden at scale.

An community user of Kubewarden (who wants to remain anonymous) provided us these information. We could use them as a starting point for that document.

Survey

How many Kubewarden's ClusterAdmissionPolicies and AdmissionPolicies do you have defined on your clusters

ClusterAdmissionPolicies: 22
AdmissionPolicies: None

How many Kubewarden's PolicyServers have you defined, what is their replica size

We decided to have 2 servers, one for context aware policies and another for all other policies. We have 15 replicas on each

How much resources (memory, CPU) are allocated to Kubewarden

We have replicas taking 300MB and 4 cores

How many admission requests do you process per minute/second/X

Some of our clusters can reach 300 requests per second (audit + webhook)

What is the latency introduced by Kubewarden, what are your constraints (this can be seen in the tracing output of Kubewaden PolicyServers)

On our biggest cluster some requests timeout at 10 seconds on the server and 2.5 seconds on the webhook. But usually, for context aware policies it can take around 500ms

What is the size of the cluster:

We have around 20 clusters

Number of nodes

Our biggest clusters have around 400 nodes

Number of Namespaces

Our biggest cluster have around 4000 namespaces

Number of Pods/RoleBinding/Ingress/other Kubernetes resources being inspected by Kubewarden

Biggest cluster:

Pods: 10000
Rolebindings: 13000
Ingresses: 12000
Deployments: 8000
Services: 13000

How often do you run the audit-scanner feature of Kubewarden

Every 4 hours

What is the helm chart configuration for the audit-scanner feature (amount of chosen parallelization?)

--parallel-namespaces
"10"
--parallel-resources
"20"
--parallel-policies
"20"
--page-size
"1000"
--disable-store

How many policies are systematically excluded from the audit-scanner

1

How long does an audit-scanner Job take

Biggest Cluster: 70 minutes