# 2. Infra-as-code (Kubespray)
## Overview
Kubespray is a composition of [Ansible](https://docs.ansible.com/) playbooks, [inventory](https://github.com/kubernetes-sigs/kubespray/blob/master/docs/ansible.md), provisioning tools, and domain knowledge for generic OS/Kubernetes cluster configuration management tasks. Kubespray provides:
* a highly available cluster
* composable attributes
* support for the most popular Linux distributions
* continuous-integration tests
## Pre-requisites
Before we can get started, we need a few prerequisites to be in place. This is what we are going to need:
1. A host with Ansible installed. [Click here](https://docs.ansible.com/) to learn more about Ansible. Find the Ansible installation details [here](https://github.com/kubernetes-sigs/kubespray/blob/master/docs/ansible.md#installing-ansible).
2. You should also set up an SSH key pair to authenticate to the Kubernetes nodes without using a password. This permits Ansible to perform optimally.
3. Few servers/hosts/VMs to serve as our targets to deploy Kubernetes. I am using Ubuntu 18.04, and my servers each have 4GB RAM and 2vCPUs. This is fine for my testing purposes, which I use to try out new things using Kubernetes. You need to be able to SSH into each of these nodes as root using the SSH key pair I mentioned above.
## Deployment Steps
The above will do the following:
* Create a new Linux User Account for use with Kubernetes on each node
* Install **Kubernetes** and **containers** on each node
* Configure the Master node
* Join the Worker nodes to the new cluster
### Install Python
Ansible needs Python to be installed on all the machines**.**
`apt-get update && apt-get install python3-pip -y`
### Disable Swap
```
sudo swapoff -a
sudo sed -i '/ swap /d' /etc/fstab
```
### Setup SSH using key-based authentication
**All the machines should be in the same network with Ubuntu or Centos installed.**
ssh key should be generated from the Bastion machine and must be copied to all the servers part of your inventory.
* Generate the ssh key `ssh-keygen -t rsa`
* Copy over the public key to all nodes.
```shell
ssh-copy-id root@
```
### Setup Ansible Controller Machine Setup Kubespray
* Clone the official repository
```
git clone https://github.com/kubernetes-incubator/kubespray.git
```
```
cd kubespray
```
* Install dependencies from `requirements.txt`
```
sudo pip install -r requirements.txt
```
* Create Inventory
```
cp -rfp inventory/sample inventory/mycluster
```
where mycluster is the custom configuration name. Replace with whatever name you would like to assign to the current cluster.
Create inventory using an inventory generator.
```
declare -a IPS=(10.67.53.158 10.67.53.159 10.67.53.160 10.67.53.161 10.67.53.162 10.67.53.163 10.67.53.164)
CONFIG_FILE=inventory/mycluster/hosts.yaml python3 contrib/inventory_builder/inventory.py ${IPS[@]}
```
Once it runs, you can see an inventory file that looks like the below:
```
all:
hosts:
node1:
ansible_host: 10.67.53.158
ip: 10.67.53.158
access_ip: 10.67.53.158
node2:
ansible_host: 10.67.53.159
ip: 10.67.53.159
access_ip: 10.67.53.159
node3:
ansible_host: 10.67.53.160
ip: 10.67.53.160
access_ip: 10.67.53.160
node4:
ansible_host: 10.67.53.161
ip: 10.67.53.161
access_ip: 10.67.53.161
node5:
ansible_host: 10.67.53.162
ip: 10.67.53.162
access_ip: 10.67.53.162
node6:
ansible_host: 10.67.53.163
ip: 10.67.53.163
access_ip: 10.67.53.163
node7:
ansible_host: 10.67.53.164
ip: 10.67.53.164
access_ip: 10.67.53.164
children:
kube-master:
hosts:
node1:
node2:
node3:
kube-node:
hosts:
node1:
node2:
node3:
node4:
node5:
node6:
node7:
etcd:
hosts:
node1:
node2:
node3:
node4:
k8s-cluster:
children:
kube-master:
kube-node:
calico-rr:
hosts: {}
```
* Review and change parameters under `inventory/mycluster/group_vars`
```
vim inventory/mycluster/group_vars/all/all.yml
...
## External LB example config
## apiserver_loadbalancer_domain_name: "elb.some.domain"
apiserver_loadbalancer_domain_name: "10.211.55.101"
loadbalancer_apiserver:
address: 10.211.55.101
port: 443
```
Deploy Kubespray with Ansible Playbook - run the playbook as Ubuntu
* The option `--become` is required for example writing SSL keys in /etc/, installing packages and interacting with various system daemons.
* **Note: Without**** ****`--become`**** ****- the playbook will fail to run!**
```
ansible-playbook -i inventory/mycluster/hosts.yaml --become --become-user=ubuntu cluster.yml
```
Kubernetes cluster will be created with three masters and four nodes using the above process.
Kube config will be generated in a .Kubefolder. The cluster can be accessible via kubeconfig.
#### **HA-Proxy**
* Install haproxy package in a haproxy machine that will be allocated for proxy
`sudo apt-get install haproxy -y`
* IPs need to be whitelisted as per the requirements in the config.
`sudo vim /etc/haproxy/haproxy.cfg`
```
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
stats timeout 30s
user haproxy
group haproxy
daemon
# Default SSL material locations
# ca-base /etc/ssl/certs
# crt-base /etc/ssl/private
# Default ciphers to use on SSL-enabled listening sockets.
# For more information, see ciphers(1SSL). This list is from:
# https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
# An alternative list with additional directives can be obtained from
# https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy
#ssl-default-bind-ciphers
#ssl-default-bind-options no-sslv3
defaults
log global
mode http
option httplog
option dontlognull
timeout connect 5000
timeout check 5000
timeout client 30000
timeout server 60000
frontend http-in
bind *:80
mode tcp
default_backend http-servers
http-request redirect scheme https unless { ssl_fc }
frontend https-in
bind *:443
mode tcp
default_backend https-servers
frontend kube-in
bind *:8383
mode tcp
timeout client 3h
#Jenkins_CD eGov_ACT eGov_Spectra
#acl network_allowed src 35.244.58.192 106.51.69.20 180.151.198.122 103.122.14.159 35.154.77.83 35.154.203.141 125.16.100.118 10.67.53.252 52.71.194.45 132.154.83.214 27.6.189.204 10.67.53.120
#tcp-request connection reject if !network_allowed
default_backend kube-servers
frontend kube2-in
bind *:6363
mode tcp
timeout client 3h
#Jenkins_CD eGov_ACT eGov_Spectra
acl network_allowed src 35.244.58.192 106.51.69.20 180.151.198.122 103.122.14.159 35.154.77.83 35.154.203.141 125.16.100.118 10.67.53.252 52.71.194.45 132.154.83.214 27.6.189.204 10.67.53.120
tcp-request connection reject if !network_allowed
default_backend kube-servers
backend http-servers
mode tcp
balance roundrobin
server srv4 10.67.53.161:32080 send-proxy
server srv5 10.67.53.162:32080 send-proxy
server srv6 10.67.53.163:32080 send-proxy
backend https-servers
mode tcp
balance roundrobin
server srv4 10.67.53.161:32443 send-proxy
server srv5 10.67.53.162:32443 send-proxy
server srv6 10.67.53.163:32443 send-proxy
backend kube-servers
mode tcp
option log-health-checks
timeout server 3h
server master1 10.67.53.158:6443 check check-ssl verify none inter 10000
server master2 10.67.53.159:6443 check check-ssl verify none inter 10000
server master3 10.67.53.160:6443 check check-ssl verify none inter 10003
balance roundrobin
```
#### **Volumes**
Iscsi volumes will be provided by the SDC team as per the requisition and the same can be used for statefulsets.
```
sudo iscsiadm -m discovery -t sendtargets -p 10.67.49.8:3260
```
**Note:** Please refer to the[ DIGIT deployment ](https://core.digit.org/guides/installation-guide/digit-deployment)documentation to deploy DIGIT services.