updates to playbooks

README.md

@@ -1,52 +1,71 @@
-## Ansible Playbooks to install an air gapped Highly Available Kubernetes cluster
-Uses Kubeadm with the --experimental-control-plane switch to install the Stacked Control Plane. This deployment pattern is well suited to single purpose, secure, infrastructure software clusters with low frequency container changes.
+# Ansible Playbooks to install an HA Kubernetes (multi-master) cluster using Kubeadm.  
 
-- Uses recently GA'd Kubeadm 1.13 HA multi-master joining features
-- Uses the Ansible Host as a short lived forward proxy to download cluster software and container images
-- No container registry is required (downloaded images are cache to the local docker engines)
+This repository provides Ansible Playbooks to install a Kubernetes HA cluster in an airgapped environment.
+- Uses recently GA'd Kubeadm HA joining features
 
 
-### Prerequisites:
+# Prerequisites:
+- Install Ansible and a forward proxy on the Ansible host
+  - Ansible:
+    for macos `brew install ansible`
+    for linux `yum install ansible`
 
-* Setup ssh access from Ansible host to Kubernetes nodes.
+- Setup ssh access from Ansible host to Kubernetes nodes.
 ```ssh-copy-id -i ~/.ssh/id_rsa.pub <user@host>```
-- Setup a local forward proxy with Internet access on the Ansible host (e.g. tinyproxy). Update the proxy environment variable details in `inventory/group_vars/all`.
 
-![Air Gapped Network Flows](./airgapped-ansible-kubernetes.png)
+# Environment preparation:
 
+Specify the Master and Workers in the `inventory/*cluster*` file:
+```
+[k8s-masters] # these are all the masters
+[k8s-workers] # these are all the worker nodes
+```
 
-### Environment preparation:
-
-Specify the Master and Worker hostnames in the `inventory/cluster` file:
-
-Update the `inventory/group_vars` sections:
+Update the `inventory/group_vars/*cluster*` section:
 - choose the desired versions for kubernetes and docker
 - setup the pod network cidr (default setup is for calico - modify in calico.yaml as well)
 - specify the version of Helm to use
 - specify the Local Storage Provisioner version
 
 
-### Install process:
+# Install a highly available kubernetes using kubeadm
+
+You can now run install-all.yaml playbook to get your cluster setup.
+You can also run the different playbooks separately for different purposes (setting up docker, masters, kubeadm, heml ...).
+
+```
+ansible-playbook -i inventory/cluster1-prod playbooks/install-all.yaml --private-key=~/.ssh/id_rsa -u %username% -v
+```
+
+# Restarting the install:
+If you need to restart the process using kubeadm reset, please use the uninstall.yaml playbook that deletes the state from all vms.
+
 
-Run install-all.yaml playbook to get your cluster fully setup. Note port 8888 below is the default for tinyproxy- adjust according to your proxy install.
-You can also run the different playbooks separately for different purposes (setting up docker, masters, kubeadm, helm ...).
+# Upgrade a highly available kubernetes using kubeadm
 
+To upgrade the kubernetes control plane run:
 ```
-ansible-playbook -i inventory/cluster playbooks/install-all.yaml --private-key=~/.ssh/id_rsa -u username --extra-vars ansible_ssh_extra_args="-R8888:localhost:8888" -v
+ansible-playbook -i inventory/cluster1-prod playbooks/upgrade-all.yaml --private-key=~/.ssh/id_rsa -u username -v
 ```
 
-### What install-all.yaml includes:
-
-- Adds the required yum repositories
-- Installs docker
-- Installs kubeadm, kubelet and kubectl
-- Initializes the first master with etcd and kubernetes-api
-- Installs Calico networking
-- Joins replica master nodes to the primary master
-- Adds the worker nodes to the cluster
-- Installs Helm & Tiller
-- Install Kubernetes Dashboard (Metrics Server support coming 02/19)
-- Installs Local Storage Provisioner (usefull for cloud native, shared nothing, stateful set apps such as Kafka, Zookeeper, Elastic)
-
-### Still to do:
-- Update Replica Master /etc/kubernetes/* file permissions after Ansible copy
+# What install-all.yaml includes:
+
+- Adding the required yum repositories
+- Installing docker
+- Installing kubeadm, kubelet and kubectl
+- Initializing the first master with etcd and kubernetes-api
+- Join replica master nodes to the primary master
+- Adding the worker nodes to the cluster
+- Installing Helm & Tiller
+- Installing Local Storage Provisioner
+- Enable Azure AD OIDC authentication
+
+# Restarting the install:
+
+If you need to restart the process using kubeadm reset, please use the uninstall.yaml playbook that deletes the state from all vms.
+
+# To sequentially drain and patch the underlying OS hosts:
+
+```
+ansible-playbook -i inventory/cluster1-prod playbooks/os-patch-updates.yaml --private-key=~/.ssh/id_rsa -u username -v
+```

ansible.cfg

@@ -5,3 +5,8 @@ inventory = inventory
 roles_path = roles
 display_skipped_hosts = false
 any_errors_fatal = true
+
+#[ssh_connection]
+#pipelining = True
+#control_path = /tmp/ansible-ssh-%%h-%%p-%%r
+#ssh_args = -C -o ControlMaster=auto -o ControlPersist=1800

inventory/cluster → inventory/cluster1-prod

@@ -1,19 +1,22 @@
 [k8s-master-primary]
-master1
+master1.domain.local
 
 [k8s-master-replicas]
-master2
-master3
+master2.domain.local
+master3.domain.local
 
 [k8s-masters:children]
 k8s-master-primary
 k8s-master-replicas
 
 [k8s-workers]
-worker1
-worker2
-worker3
+worker1.domain.local
+worker2.domain.local
+worker3.domain.local
 
 [k8s-nodes:children]
 k8s-masters
 k8s-workers
+
+[cluster1-prod:children]
+k8s-nodes

inventory/cluster2-nonprod

@@ -0,0 +1,22 @@
+[k8s-master-primary]
+master1.domain.local
+
+[k8s-master-replicas]
+master2.domain.local
+master3.domain.local
+
+[k8s-masters:children]
+k8s-master-primary
+k8s-master-replicas
+
+[k8s-workers]
+worker1.domain.local
+worker2.domain.local
+worker3.domain.local
+
+[k8s-nodes:children]
+k8s-masters
+k8s-workers
+
+[cluster2-nonprod:children]
+k8s-nodes

inventory/group_vars/cluster1-prod/all

@@ -0,0 +1,58 @@
+---
+
+docker_version: 18.09.6
+kubernetes_version: v1.15.0
+kubeadm_version: 1.15.0
+
+# kubernetes API load balanced VIP for HA installations
+kubernetes_loadbalanced_api_dns: k8s-api01.domain.local
+kubernetes_cluster_label: cluster1-prod
+
+# Container registry01
+container_registry: container-registry01.nonprod.domain.local
+
+# Docker Daemon configuration
+docker_ce_daemon_options:
+    exec-opts: [ "native.cgroupdriver=systemd" ]
+    log-driver: json-file
+    log-opts:
+        max-size: "100m"
+        max-file: "7"
+    storage-driver: overlay2
+    storage-opts: [ "overlay2.override_kernel_check=true" ]
+
+# Kubernetes Kubeadm Cluster Configuration
+kubeadm_config_options:
+    apiVersion: kubeadm.k8s.io/v1beta1
+    kind: ClusterConfiguration
+    kubernetesVersion: "{{ kubernetes_version }}"
+    apiServer:
+        certSANs:
+            - "{{ kubernetes_loadbalanced_api_dns }}"
+    controlPlaneEndpoint: "{{ kubernetes_loadbalanced_api_dns }}:6443"
+    networking:
+        podSubnet: 10.244.0.0/16
+    imageRepository: "{{ container_registry }}/keystone"
+    useHyperKubeImage: true
+    clusterName: "{{ kubernetes_cluster_label }}"
+    etcd:
+        local:
+            imageRepository: "{{ container_registry }}/keystone"
+
+
+# Addon Container Images
+tiller_image: "{{ container_registry }}/keystone/tiller:v2.12.1"
+
+# Filebeat Kafka Topic Name
+filebeat_app_id: kubernetes-logs-prod
+
+# Metricbeat Kafka Topic Name
+metricbeat_app_id: metricbeat-kubernetes-prod
+
+# Azure AD Admin group and AppID
+aad_admin_groupname: XXXXX
+aad_admin_groupid: XXXXX
+
+# Azure AD User group and AppID
+aad_user_groupname: XXXXX
+aad_user_groupid: XXXXX

inventory/group_vars/cluster2-nonprod/all

@@ -0,0 +1,58 @@
+---
+
+docker_version: 18.09.6
+kubernetes_version: v1.15.0
+kubeadm_version: 1.15.0
+
+# kubernetes API load balanced VIP for HA installations
+kubernetes_loadbalanced_api_dns: k8s-api02.nonprod.domain.local
+kubernetes_cluster_label: cluster2-nonprod
+
+# Container registry01
+container_registry: container-registry01.nonprod.domain.local
+
+# Docker Daemon configuration
+docker_ce_daemon_options:
+    exec-opts: [ "native.cgroupdriver=systemd" ]
+    log-driver: json-file
+    log-opts:
+        max-size: "100m"
+        max-file: "7"
+    storage-driver: overlay2
+    storage-opts: [ "overlay2.override_kernel_check=true" ]
+
+# Kubernetes Kubeadm Cluster Configuration
+kubeadm_config_options:
+    apiVersion: kubeadm.k8s.io/v1beta1
+    kind: ClusterConfiguration
+    kubernetesVersion: "{{ kubernetes_version }}"
+    apiServer:
+        certSANs:
+            - "{{ kubernetes_loadbalanced_api_dns }}"
+    controlPlaneEndpoint: "{{ kubernetes_loadbalanced_api_dns }}:6443"
+    networking:
+        podSubnet: 10.244.0.0/16
+    imageRepository: "{{ container_registry }}/keystone"
+    useHyperKubeImage: true
+    clusterName: "{{ kubernetes_cluster_label }}"
+    etcd:
+        local:
+            imageRepository: "{{ container_registry }}/keystone"
+
+
+# Addon Container Images
+tiller_image: "{{ container_registry }}/keystone/tiller:v2.12.1"
+
+# Filebeat Kafka Topic Name
+filebeat_app_id: kubernetes-logs-nonprod
+
+# Metricbeat Kafka Topic Name
+metricbeat_app_id: metricbeat-kubernetes-nonprod
+
+# Azure AD Admin group and AppID
+aad_admin_groupname: XXXXX
+aad_admin_groupid: XXXXX
+
+# Azure AD User group and AppID
+aad_user_groupname: XXXXX
+aad_user_groupid: XXXXX

playbooks/aad-authentication.yaml

@@ -0,0 +1,4 @@
+- hosts: k8s-master-primary
+  become: yes
+  roles:
+    - aad-authentication

playbooks/kafka-charts.yaml → playbooks/elastic-search.yaml

@@ -1,4 +1,4 @@
 - hosts: k8s-master-primary
   become: yes
   roles:
-    - kafka-charts
+    - elastic-search

playbooks/install-all.yaml

@@ -2,10 +2,13 @@
 - import_playbook: repos.yaml
 - import_playbook: docker.yaml
 - import_playbook: kubeadm-prep.yaml
+#- import_playbook: kubeadm-singlenode-cluster.yaml
 - import_playbook: kubeadm-init-master.yaml
 - import_playbook: calico.yaml
 - import_playbook: kubeadm-join-masters.yaml
 - import_playbook: kubeadm-join-workers.yaml
 - import_playbook: helm-install.yaml
 - import_playbook: local-storage-provisioner.yaml
-- import_playbook: kubernetes-dashboard.yaml
+- import_playbook: kubernetes-dashboard.yaml
+- import_playbook: elastic-search.yaml
+- import_playbook: aad-authentication.yaml

playbooks/install_docker.yaml

@@ -0,0 +1,29 @@
+- name: Install Docker-CE engine
+  yum: 
+    name: docker-ce-{{ docker_version }}
+    state: installed
+    enablerepo: docker-ce  
+
+- name: Create directory for trusted registry
+  file:
+    path: /etc/docker/certs.d/container-registry01.nonprod.domain.local
+    state: directory
+
+- name: Copy registry CA certificate
+  template:
+    src: container-registry01.nonprod.domain.local-ca.crt
+    dest: /etc/docker/certs.d/container-registry01.nonprod.domain.local/ca.crt
+    force: yes
+
+- name: Configure additional engine options
+  copy:
+    content: "{{ docker_ce_daemon_options | to_nice_json }}"
+    dest: /etc/docker/daemon.json
+    mode: 0644
+  when: docker_ce_daemon_options is defined
+
+- name: Enable docker service
+  systemd:
+    name: docker
+    state: restarted
+    enabled: yes

playbooks/kubeadm-singlenode-cluster.yaml

@@ -0,0 +1,4 @@
+- hosts: k8s-master-primary
+  become: yes
+  roles:
+    - kubeadm-singlenode-cluster