Deployment on SDC
Running Kubernetes on-premise gives a cloud-native experience on SDC when it comes to deploying DIGIT.
Whether States have their own on-premise data centre or have decided to forego the various managed cloud solutions, there are a few things one should know when getting started with on-premise K8s.
One should be familiar with Kubernetes and the control plane consists of the Kube-apiserver, Kube-scheduler, Kube-controller-manager and an ETCD datastore. For managed cloud solutions like Google’s Kubernetes Engine (GKE) or Azure’s Kubernetes Service (AKS), it also includes the cloud-controller-manager. This is the component that connects the cluster to external cloud services to provide networking, storage, authentication, and other support features.
To successfully deploy a bespoke Kubernetes cluster and achieve a cloud-like experience on SDC, one needs to replicate all the same features you get with a managed solution. At a high level, this means that we probably want to:
Automate the deployment process
Choose a networking solution
Choose a right storage solution
Handle security and authentication
The subsequent sections look at each of these challenges individually, and provide enough of a context required to help in getting started.
Using a tool like Ansible can make deploying Kubernetes clusters on-premise trivial.
When deciding to manage your own Kubernetes clusters, we need to set up a few proofs-of-concept (PoC) clusters to learn how everything works, perform performance and conformance tests, and try out different configuration options.
After this phase, automating the deployment process is an important if not necessary step to ensure consistency across any clusters you build. For this, you have a few options, but the most popular are:
kubeadm: a low-level tool that helps you bootstrap a minimum viable Kubernetes cluster that conforms to best practices
kubespray: an Ansible playbook that helps deploy production-ready clusters
If you already using Ansible, Kubespray is a great option, otherwise, we recommend writing automation around Kubeadm using your preferred playbook tool after using it a few times. This will also increase your confidence and knowledge of Kubernetes.
Deploy DIGIT using Kubespray
Kubespray is a composition of Ansible playbooks, inventory, provisioning tools, and domain knowledge for generic OS/Kubernetes cluster configuration management tasks. Kubespray provides:
a highly available cluster
composable attributes
support for most popular Linux distributions
continuous-integration tests
Before we can get started, we need a few prerequisites to be in place. This is what we are going to need:
A host with Ansible installed. Click here to learn more about Ansible. Find the Ansible installation details here.
You should also set up an SSH key pair to authenticate to the Kubernetes nodes without using a password. This permits Ansible to perform optimally.
Few servers/hosts/VMs to serve as our targets to deploy Kubernetes. I am using Ubuntu 18.04, and my servers each have 4GB RAM and 2vCPUs. This is fine for my testing purposes, which I use to try out new things using Kubernetes. You need to be able to SSH into each of these nodes as root using the SSH key pair I mentioned above.
The above will do the following:
Create a new Linux User Account for use with Kubernetes on each node
Install Kubernetes and containers on each node
Configure the Master node
Join the Worker nodes to the new cluster
Ansible needs Python to be installed on all the machines.
apt-get update && apt-get install python3-pip -y
All the machines should be in the same network with Ubuntu or Centos installed.
ssh key should be generated from the Bastion machine and must be copied to all the servers part of your inventory.
Generate the ssh key ssh-keygen -t rsa
Copy over the public key to all nodes.
Clone the official repository
Install dependencies from requirements.txt
Create Inventory
where mycluster is the custom configuration name. Replace with whatever name you would like to assign to the current cluster.
Create inventory using an inventory generator.
Once it runs, you can see an inventory file that looks like the below:
Review and change parameters under inventory/mycluster/group_vars
Deploy Kubespray with Ansible Playbook - run the playbook as Ubuntu
The option --become is required for example writing SSL keys in /etc/, installing packages and interacting with various system daemons.
Note: Without --become - the playbook will fail to run!
Kubernetes cluster will be created with three masters and four nodes using the above process.
Kube config will be generated in a .Kubefolder. The cluster can be accessible via kubeconfig.
Install haproxy package in a haproxy machine that will be allocated for proxy
sudo apt-get install haproxy -y
IPs need to be whitelisted as per the requirements in the config.
sudo vim /etc/haproxy/haproxy.cfg
Iscsi volumes will be provided by the SDC team as per the requisition and the same can be used for statefulsets.
Note: Please refer to the DIGIT deployment documentation to deploy DIGIT services.
Steps to setup CI/CD on SDC
Kubespray is a composition of Ansible playbooks, inventory, provisioning tools, and domain knowledge for generic OS/Kubernetes cluster configuration management tasks. Kubespray provides:
a highly available cluster
composable attributes
support for most popular Linux distributions
continuous-integration tests
Fork the repos below to your GitHub Organization account
Go lang (version 1.13.X)
Install kubectl on your local machine to interact with the Kubernetes cluster.
Install Helm to help package the services along with the configurations, environment, secrets, etc into Kubernetes manifests.
One Bastion machine to run Kubespray
HA-PROXY machine which acts as a load balancer with Public IP. (CPU: 2Core , Memory: 4Gb)
one machine which acts as a master node. (CPU: 2Core , Memory: 4Gb)
one machine which acts as a worker node. (CPU: 8Core , Memory: 16Gb)
ISCSI volumes for persistence volume. (number of quantity: 2 )
kaniko-cache-claim:- 10Gb
Jenkins home:- 100Gb
Kubernetes nodes
Ubuntu 18.04
SSH
Privileged user
Python
Run and follow instructions on all nodes.
Ansible needs Python to be installed on all the machines.
apt-get update && apt-get install python3-pip -y
All the machines should be in the same network with ubuntu or centos installed.
ssh key should be generated from the Bastion machine and must be copied to all the servers part of your inventory.
Generate the ssh key ssh-keygen -t rsa
Copy over the public key to all nodes.
Clone the official repository
Install dependencies from requirements.txt
Create Inventory
where mycluster is the custom configuration name. Replace with whatever name you would like to assign to the current cluster.
Create inventory using an inventory generator.
Once it runs, you can see an inventory file that looks like the below:
Review and change parameters under inventory/mycluster/group_vars
Deploy Kubespray with Ansible Playbook - run the playbook as Ubuntu
The option --become is required, for example writing SSL keys in /etc/, installing packages and interacting with various system daemons.
Note: Without --become - the playbook will fail to run!
Kubernetes cluster will be created with three masters and four nodes with the above process.
Kube config will be generated in a .Kubefolder. The cluster can be accessible via kubeconfig.
Install haproxy package in a haproxy machine that will be allocated for proxy
sudo apt-get install haproxy -y
IPs need to be whitelisted as per the requirements in the config.
sudo vim /etc/haproxy/haproxy.cfg
Iscsi volumes will be provided by the SDC team as per the requisition and the same can be used for statefulsets.
Refer to the doc here.