Bootstrapping a cluster involves the following steps:
- The bootstrap machine boots and starts hosting the remote resources required for the master machines to boot.
- The master machines fetch the remote resources from the bootstrap machine and finish booting.
- The master machines use the bootstrap machine to form an etcd cluster.
- The bootstrap machine starts a temporary Kubernetes control plane using the new etcd cluster.
- The temporary control plane schedules the production control plane to the master machines.
- The temporary control plane shuts down and passes control to the production control plane.
- The bootstrap machine injects OpenShift Container Platform components into the production control plane.
- The installation program shuts down the bootstrap machine. It can now be removed from the api and api-int loadbalancers.
- The control plane sets up the worker nodes.
- The control plane installs additional services in the form of a set of Operators.
The result of this bootstrapping process is a fully running OpenShift Container Platform cluster. The cluster then downloads and configures remaining components needed for the day-to-day operation, including the creation of worker machines in supported environments.
On supported platforms, the installer is capable of provisioning the underlying infrastructure for the cluster. The installer programmatically creates all portions of the networking, machines, and operating systems required to support the cluster. This is called the Installer Provisioned Infrastructure (IPI) pattern. Think of it as best-practice reference architecture implemented in code. It is recommended that most users make use of this functionality to avoid having to provision their own infrastructure. The installer will create and destroy the infrastructure components it needs to be successful over the life of the cluster.
$ openshift-install create cluster
? SSH Public Key /Users/ncolon/.ssh/openshift_rsa.pub
? Platform azure
? Region eastus2
? Base Domain azure.faststart.tk
? Cluster Name vegas
? Pull Secret [? for help] *********************************************************** <...>
INFO Creating infrastructure resources...
INFO Waiting up to 30m0s for the Kubernetes API at https://api.vegas.azure.faststart.tk:6443...
INFO API v1.14.6+2e5ed54 up
INFO Waiting up to 30m0s for bootstrapping to complete...
INFO Destroying the bootstrap resources...
INFO Waiting up to 30m0s for the cluster at https://api.vegas.azure.faststart.tk:6443 to initialize...
INFO Waiting up to 10m0s for the openshift-console route to be created...
INFO Install complete!
INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/Users/ncolon/auth/kubeconfig'
INFO Access the OpenShift web-console here: https://console-openshift-console.apps.vegas.azure.faststart.tk
INFO Login to the console with user: kubeadmin, password: NDNau-9S9Ab-25Re2-6H3eQThe SSH key specified above needs to be a passwordless ssh key. You can generate it with the command
$ ssh-keygen -t rsa -b 4096 -N '' -f ~/.ssh/openshift_rsaOnce the installation process is complete, you will have a link to the openshift console, as well as credentials to log.
For other platforms or in scenarios where installer provisioned infrastructure would be incompatible, the installer can stop short of creating the infrastructure, and allow the platform administrator to provision their own using the cluster assets generated by the install tool. This pattern is called User Provisioned Infrastructure (UPI). Once the infrastructure has been created, OpenShift 4 is installed, maintaining its ability to support automated operations and over-the-air platform updates.
$ openshift-install --dir=/tmp/upi-cluster create install-config
? SSH Public Key /Users/ncolon/.ssh/openshift_rsa.pub
? Platform azure
? Region eastus2
? Base Domain azure.faststart.tk
? Cluster Name vegas
? Pull Secret [? for help] *********************************************************** <...>
$ cat install-config.yaml
apiVersion: v1
baseDomain: azure.faststart.tk
compute:
- hyperthreading: Enabled
name: worker
platform: {<CLOUD SPECIFIC>}
replicas: 3
controlPlane:
hyperthreading: Enabled
name: master
platform: {<CLOUD SPECIFIC>}
replicas: 3
metadata:
creationTimestamp: null
name: vegas
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
machineCIDR: 10.0.0.0/16
networkType: OpenShiftSDN
serviceNetwork:
- 172.30.0.0/16
platform: {<CLOUD SPECIFIC>}
pullSecret: '{"auths": {<...>}}'
sshKey: |
ssh-rsa <XXXXXXXXXXXXXXXXXXXXXXXXXX> user@workstationFor compute.platform, controlPlane.platform and platform sections, you can specify cloud provider information, such as the flavor of your master and worker nodes, OS disk storage, availability zones to deploy to, as well as any specific networking configuration. A list of possible parameters for your install-config.yaml file can be found here. Platform specific samples can also be found in the public OpenShift documentation.
$ openshift-install --dir=/tmp/upi-cluster/ create manifests
INFO Consuming "Install Config" from target directory
$ ls -la /tmp/upi-cluster/
total 2952
drwxr-xr-x 6 ncolon wheel 192 Feb 11 18:57 .
drwxrwxrwt 29 root wheel 928 Feb 11 18:57 ..
-rw-r--r-- 1 ncolon wheel 142255 Feb 11 18:57 .openshift_install.log
-rw-r--r-- 1 ncolon wheel 1310080 Feb 11 18:57 .openshift_install_state.json
drwxr-xr-x 28 ncolon wheel 896 Feb 11 18:57 manifests
drwxr-xr-x 15 ncolon wheel 480 Feb 11 18:57 openshiftThis command will remove the install-config.yaml created in Step 1. If needed, you can recreate the install-config.yaml file by running the openshift-install command from Step 1. It will also create 2 directories in your installation path, openshift and manifests
The openshift directory holds information necesary to create the control plane nodes machines, credentials to interface with your cloud provider, and other configuration required to create your worker MachineSets. There should be as many worker MachineSets configuration files as availability zones in your region. If you want to create more MachineSets for your installation (for example, for Storage nodes), you can use the existing ones as a template.
The manifests directory holds configuration parameters for your cluster. If you want to modify objects related to your cluster, you can do so here. You can, for example, modify the networking parameters of your cluster by adding a file called cluster-network-03-config.yaml with the following YAML.
apiVersion: operator.openshift.io/v1
kind: Network
metadata:
name: cluster
spec:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
serviceNetwork:
- 172.30.0.0/16
defaultNetwork:
type: OpenShiftSDN
openshiftSDNConfig:
mode: NetworkPolicy
mtu: 1450
vxlanPort: 4789The files are executed in order, so cluster-network-03-config.yaml would get executed after cluster-network-02-config.yaml
The next step involves generating the ignition config files that are passed along to your CoreOS nodes for configuration. As you create each cluster node role, you must pass the corresponding ignition configuration file. These files are meant to be hosted on a webserver or other web endpoint that your cluster nodes can access at boot time, and are only necesary throughout the bootstrap process. Once the cluster is properly deployed, adding additional worker nodes is handled via the api-int loadbalancer endpoint.
The Ignition config files that the installation program generates contain certificates that expire after 24 hours. You must complete your cluster installation and keep the cluster running for 24 hours in a non-degraded state to ensure that the first certificate rotation has finished.
$ openshift-install --dir=/tmp/upi-cluster/ create ignition-configs
INFO Consuming "Common Manifests" from target directory
INFO Consuming "Worker Machines" from target directory
INFO Consuming "Master Machines" from target directory
INFO Consuming "Openshift Manifests" from target directory
$ ls -la /tmp/upi-cluster/
total 3600
drwxr-xr-x 9 ncolon wheel 288 Feb 11 19:07 .
drwxrwxrwt 29 root wheel 928 Feb 11 19:07 ..
-rw-r--r-- 1 ncolon wheel 175288 Feb 11 19:07 .openshift_install.log
-rw-r--r-- 1 ncolon wheel 1310080 Feb 11 19:07 .openshift_install_state.json
drwxr-xr-x 4 ncolon wheel 128 Feb 11 19:07 auth
-rw-r--r-- 1 ncolon wheel 318820 Feb 11 19:07 bootstrap.ign
-rw-r--r-- 1 ncolon wheel 1834 Feb 11 19:07 master.ign
-rw-r--r-- 1 ncolon wheel 137 Feb 11 19:07 metadata.json
-rw-r--r-- 1 ncolon wheel 1834 Feb 11 19:07 worker.ignBoot up the compute nodes. After about 30-45 minutes, your cluster should be up and running
$ oc get nodes
NAME STATUS ROLES AGE VERSION
vegas-6ric0-master-0 Ready master 6h45m v1.16.2
vegas-6ric0-master-1 Ready master 6h45m v1.16.2
vegas-6ric0-master-2 Ready master 6h45m v1.16.2
vegas-6ric0-worker-eastus21-cllqw Ready worker 6h36m v1.16.2
vegas-6ric0-worker-eastus22-5w247 Ready worker 6h35m v1.16.2
vegas-6ric0-worker-eastus23-kjmxm Ready worker 6h35m v1.16.2
$ oc get clusteroperators
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE
authentication 4.3.0 True False False 6h27m
cloud-credential 4.3.0 True False False 6h45m
cluster-autoscaler 4.3.0 True False False 6h35m
console 4.3.0 True False False 6h30m
dns 4.3.0 True False False 6h41m
image-registry 4.3.0 True False False 6h34m
ingress 4.3.0 True False False 6h34m
insights 4.3.0 True False False 6h42m
kube-apiserver 4.3.0 True False False 6h39m
kube-controller-manager 4.3.0 True False False 6h39m
kube-scheduler 4.3.0 True False False 6h40m
machine-api 4.3.0 True False False 6h41m
machine-config 4.3.0 True False False 6h36m
marketplace 4.3.0 True False False 6h37m
monitoring 4.3.0 True False False 6h32m
network 4.3.0 True False False 6h41m
node-tuning 4.3.0 True False False 6h37m
openshift-apiserver 4.3.0 True False False 6h36m
openshift-controller-manager 4.3.0 True False False 6h40m
openshift-samples 4.3.0 True False False 6h35m
operator-lifecycle-manager 4.3.0 True False False 6h41m
operator-lifecycle-manager-catalog 4.3.0 True False False 6h41m
operator-lifecycle-manager-packageserver 4.3.0 True False False 6h37m
service-ca 4.3.0 True False False 6h42m
service-catalog-apiserver 4.3.0 True False False 6h37m
service-catalog-controller-manager 4.3.0 True False False 6h37m
storage 4.3.0 True False False 6h36mOnce you're done with your cluster, you may wish to delete it. To destroy the cluster, the openshift-install binary provides parameters to do so.
$ openshift-install --dir=/tmp/upi-cluster/ destroy cluster
INFO deleted record=api.vegas
INFO deleted record="*.apps.vegas"
INFO deleted resource group=vegas-26g2z-rg
INFO deleted appID=b761ade6-d7c6-439a-ad8e-a8c112e1fd48
INFO deleted appID=155025a6-8d55-42d1-8293-3f674c10f23e
INFO deleted appID=f5d4659f-776b-4f66-adee-576917b4eb67Destroying the cluster in this way will ensure that no artifacts are left behind, consuming resources and potentially incurring in cloud provider costs.
You can mirror the contents of the OpenShift Container Platform registry and the images that are required to generate the installation program. If you're performing a disconnected installation, you must install your cluster using the UPI process described above.
The mirror registry is a key component that is required to complete an installation in a restricted network. You can create this mirror on a bastion host, which can access both the internet and your closed network, or by using other methods that meet your restrictions.
Because of the way that OpenShift Container Platform verifies integrity for the release payload, the image references in your local registry are identical to the ones that are hosted by Red Hat on Quay.io. During the bootstrapping process of installation, the images must have the same digests no matter which repository they are pulled from. To ensure that the release payload is identical, you mirror the images to your local repository.
RedHat provides step by step instructions on creating an installation environment for installation in a restricted network. The process can be summarized as:
-
Create a mirror registry. You can build your own using podman and a generic web server or you can use a packaged image regustry solution like Harbor or Quay which provides image scanning and other advanced functionality.
You can also use Amazon ECR, Azure ACR and Google GCR as your image registry. Each cloud provider provides mechanisms so that communication to their image registry solution happens though their network backbone and not through the public internet.
-
Once your image registry is created, you need to update your pull-secret file with its authentication details.
"auths": { ... "<local_registry_host_name>:<local_registry_host_port>": { "auth": "<credentials>", "email": "you@example.com" }, ...
-
You then need to clone the official OpenShift 4.x quay.io registry. The
occommand line tool provides functionality that will pull all required artifacts from this external repository.
$ export OCP_RELEASE=<release_version>
$ export LOCAL_REGISTRY='<local_registry_host_name>:<local_registry_host_port>'
$ export LOCAL_REPOSITORY='<repository_name>'
$ export PRODUCT_REPO='openshift-release-dev'
$ export LOCAL_SECRET_JSON='<path_to_pull_secret>'
$ export RELEASE_NAME="ocp-release"
$ oc adm -a ${LOCAL_SECRET_JSON} release mirror \
--from=quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE} \
--to=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} \
--to-release-image=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}If your registry is not protected with TLS, add the --insecure flag to the oc command. Once this command completes, it will output several snippets that you must add to the installation configuration, and to your OpenShift cluster to provide updates.
$ export OCP_RELEASE="4.3.0-x86_64"
$ export LOCAL_REGISTRY="ncm-mbpr.local:80"
$ export LOCAL_REPOSITORY="openshift4/ocp4"
$ export PRODUCT_REPO='openshift-release-dev'
$ export LOCAL_SECRET_JSON='./pull-secret'
$ export RELEASE_NAME="ocp-release"
$ oc adm -a ${LOCAL_SECRET_JSON} release mirror --insecure \
--from=quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE} \
--to=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} \
--to-release-image=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}
...
info: Mirroring completed in 8m46.5s (10.38MB/s)
Success
Update image: ncm-mbpr.local:80/openshift4/ocp4:4.3.0-x86_64
Mirror prefix: ncm-mbpr.local:80/openshift4/ocp4To use the new mirrored repository to install, add the following section to the bottom of your install-config.yaml:
imageContentSources:
- mirrors:
- ncm-mbpr.local:80/openshift4/ocp4
source: quay.io/openshift-release-dev/ocp-release
- mirrors:
- ncm-mbpr.local:80/openshift4/ocp4
source: quay.io/openshift-release-dev/ocp-v4.0-art-dev$ cat install-config.yaml
apiVersion: v1
baseDomain: azure.faststart.tk
compute: { ... }
controlPlane: { ... }
metadata:
creationTimestamp: null
name: vegas
networking: { ... }
platform: { ... }
pullSecret: '{"auths": {<...>}}'
sshKey: |
ssh-rsa <XXXXXXXXXXXXXXXXXXXXXXXXXX> user@workstation
imageContentSources:
- mirrors:
- ncm-mbpr.local:80/openshift4/ocp4
source: quay.io/openshift-release-dev/ocp-release
- mirrors:
- ncm-mbpr.local:80/openshift4/ocp4
source: quay.io/openshift-release-dev/ocp-v4.0-art-devTo use the new mirrored repository for upgrades, use the following to create an ImageContentSourcePolicy:
apiVersion: operator.openshift.io/v1alpha1
kind: ImageContentSourcePolicy
metadata:
name: example
spec:
repositoryDigestMirrors:
- mirrors:
- ncm-mbpr.local:80/openshift4/ocp4
source: quay.io/openshift-release-dev/ocp-release
- mirrors:
- ncm-mbpr.local:80/openshift4/ocp4
source: quay.io/openshift-release-dev/ocp-v4.0-art-devThe prefered method for installing your cluster should be through IPI. However, if you need to customize your cluster in a way thats not provided by IPI, the team at IBM Garage Solution Engineering has developed automation that can assist you in deploying your cluster as described in this reference architecture document.
Implementations are avaialble for AWS, Azure and GCP. Each repository has information on what parameters need to be provided, as well as what parameters can be changed.