# 2. Infra-as-code (Kubespray)
## Overview
Kubespray is a composition of [Ansible](https://docs.ansible.com/) playbooks, [inventory](https://github.com/kubernetes-sigs/kubespray/blob/master/docs/ansible.md), provisioning tools, and domain knowledge for generic OS/Kubernetes cluster configuration management tasks. Kubespray provides:
* a highly available cluster
* composable attributes
* support for the most popular Linux distributions
* continuous-integration tests
## Pre-requisites
Before we can get started, we need a few prerequisites to be in place. This is what we are going to need:
1. A host with Ansible installed. [Click here](https://docs.ansible.com/) to learn more about Ansible. Find the Ansible installation details [here](https://github.com/kubernetes-sigs/kubespray/blob/master/docs/ansible.md#installing-ansible).
2. You should also set up an SSH key pair to authenticate to the Kubernetes nodes without using a password. This permits Ansible to perform optimally.
3. Few servers/hosts/VMs to serve as our targets to deploy Kubernetes. I am using Ubuntu 18.04, and my servers each have 4GB RAM and 2vCPUs. This is fine for my testing purposes, which I use to try out new things using Kubernetes. You need to be able to SSH into each of these nodes as root using the SSH key pair I mentioned above.
## Deployment Steps
The above will do the following:
* Create a new Linux User Account for use with Kubernetes on each node
* Install **Kubernetes** and **containers** on each node
* Configure the Master node
* Join the Worker nodes to the new cluster
### Install Python
Ansible needs Python to be installed on all the machines**.**
`apt-get update && apt-get install python3-pip -y`
### Disable Swap
```
sudo swapoff -a
sudo sed -i '/ swap /d' /etc/fstab
```
### Setup SSH using key-based authentication
**All the machines should be in the same network with Ubuntu or Centos installed.**
ssh key should be generated from the Bastion machine and must be copied to all the servers part of your inventory.
* Generate the ssh key `ssh-keygen -t rsa`
* Copy over the public key to all nodes.
```shell
ssh-copy-id root@
```
### Setup Ansible Controller Machine Setup Kubespray
* Clone the official repository
```
git clone https://github.com/kubernetes-incubator/kubespray.git
```
```
cd kubespray
```
* Install dependencies from `requirements.txt`
```
sudo pip install -r requirements.txt
```
* Create Inventory
```
cp -rfp inventory/sample inventory/mycluster
```
where mycluster is the custom configuration name. Replace with whatever name you would like to assign to the current cluster.
Create inventory using an inventory generator.
```
declare -a IPS=(10.67.53.158 10.67.53.159 10.67.53.160 10.67.53.161 10.67.53.162 10.67.53.163 10.67.53.164)
CONFIG_FILE=inventory/mycluster/hosts.yaml python3 contrib/inventory_builder/inventory.py ${IPS[@]}
```
Once it runs, you can see an inventory file that looks like the below:
```
all:
hosts:
node1:
ansible_host: 10.67.53.158
ip: 10.67.53.158
access_ip: 10.67.53.158
node2:
ansible_host: 10.67.53.159
ip: 10.67.53.159
access_ip: 10.67.53.159
node3:
ansible_host: 10.67.53.160
ip: 10.67.53.160
access_ip: 10.67.53.160
node4:
ansible_host: 10.67.53.161
ip: 10.67.53.161
access_ip: 10.67.53.161
node5:
ansible_host: 10.67.53.162
ip: 10.67.53.162
access_ip: 10.67.53.162
node6:
ansible_host: 10.67.53.163
ip: 10.67.53.163
access_ip: 10.67.53.163
node7:
ansible_host: 10.67.53.164
ip: 10.67.53.164
access_ip: 10.67.53.164
children:
kube-master:
hosts:
node1:
node2:
node3:
kube-node:
hosts:
node1:
node2:
node3:
node4:
node5:
node6:
node7:
etcd:
hosts:
node1:
node2:
node3:
node4:
k8s-cluster:
children:
kube-master:
kube-node:
calico-rr:
hosts: {}
```
* Review and change parameters under `inventory/mycluster/group_vars`
```
vim inventory/mycluster/group_vars/all/all.yml
...
## External LB example config
## apiserver_loadbalancer_domain_name: "elb.some.domain"
apiserver_loadbalancer_domain_name: "10.211.55.101"
loadbalancer_apiserver:
address: 10.211.55.101
port: 443
```
Deploy Kubespray with Ansible Playbook - run the playbook as Ubuntu
* The option `--become` is required for example writing SSL keys in /etc/, installing packages and interacting with various system daemons.
* **Note: Without**** ****`--become`**** ****- the playbook will fail to run!**
```
ansible-playbook -i inventory/mycluster/hosts.yaml --become --become-user=ubuntu cluster.yml
```
Kubernetes cluster will be created with three masters and four nodes using the above process.
Kube config will be generated in a .Kubefolder. The cluster can be accessible via kubeconfig.
#### **HA-Proxy**
* Install haproxy package in a haproxy machine that will be allocated for proxy
`sudo apt-get install haproxy -y`
* IPs need to be whitelisted as per the requirements in the config.
`sudo vim /etc/haproxy/haproxy.cfg`
```
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
stats timeout 30s
user haproxy
group haproxy
daemon
# Default SSL material locations
# ca-base /etc/ssl/certs
# crt-base /etc/ssl/private
# Default ciphers to use on SSL-enabled listening sockets.
# For more information, see ciphers(1SSL). This list is from:
# https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
# An alternative list with additional directives can be obtained from
# https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy
#ssl-default-bind-ciphers
#ssl-default-bind-options no-sslv3
defaults
log global
mode http
option httplog
option dontlognull
timeout connect 5000
timeout check 5000
timeout client 30000
timeout server 60000
frontend http-in
bind *:80
mode tcp
default_backend http-servers
http-request redirect scheme https unless { ssl_fc }
frontend https-in
bind *:443
mode tcp
default_backend https-servers
frontend kube-in
bind *:8383
mode tcp
timeout client 3h
#Jenkins_CD eGov_ACT eGov_Spectra
#acl network_allowed src 35.244.58.192 106.51.69.20 180.151.198.122 103.122.14.159 35.154.77.83 35.154.203.141 125.16.100.118 10.67.53.252 52.71.194.45 132.154.83.214 27.6.189.204 10.67.53.120
#tcp-request connection reject if !network_allowed
default_backend kube-servers
frontend kube2-in
bind *:6363
mode tcp
timeout client 3h
#Jenkins_CD eGov_ACT eGov_Spectra
acl network_allowed src 35.244.58.192 106.51.69.20 180.151.198.122 103.122.14.159 35.154.77.83 35.154.203.141 125.16.100.118 10.67.53.252 52.71.194.45 132.154.83.214 27.6.189.204 10.67.53.120
tcp-request connection reject if !network_allowed
default_backend kube-servers
backend http-servers
mode tcp
balance roundrobin
server srv4 10.67.53.161:32080 send-proxy
server srv5 10.67.53.162:32080 send-proxy
server srv6 10.67.53.163:32080 send-proxy
backend https-servers
mode tcp
balance roundrobin
server srv4 10.67.53.161:32443 send-proxy
server srv5 10.67.53.162:32443 send-proxy
server srv6 10.67.53.163:32443 send-proxy
backend kube-servers
mode tcp
option log-health-checks
timeout server 3h
server master1 10.67.53.158:6443 check check-ssl verify none inter 10000
server master2 10.67.53.159:6443 check check-ssl verify none inter 10000
server master3 10.67.53.160:6443 check check-ssl verify none inter 10003
balance roundrobin
```
#### **Volumes**
Iscsi volumes will be provided by the SDC team as per the requisition and the same can be used for statefulsets.
```
sudo iscsiadm -m discovery -t sendtargets -p 10.67.49.8:3260
```
**Note:** Please refer to the[ DIGIT deployment ](https://core.digit.org/guides/installation-guide/digit-deployment)documentation to deploy DIGIT services.
---
# Agent Instructions: Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter:
```
GET https://docs.digit.org/platform/guides/installation-guide/infrastructure-setup/sdc/2.-infra-as-code-kubespray.md?ask=
```
The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.