Baremetal deployment

Kubernetes cluster deployment

The recommended way to provision a baremetal Kubernetes cluster is Kubespay.

Make sure all of your future cluster nodes are accessible and have your SSH key provisioned.

To provision the cluster, perform the following steps:

  • git clone git@github.com:kubernetes-sigs/kubespray.git
  • cd kubespray
  • git checkout v2.10.4
  • pip3 install -r requirements.txt
  • cp -rfp inventory/sample inventory/*CLUSTER_NAME*
  • declare -a IPS=(*IPS_OF_MACHINES*)
  • CONFIG_FILE=inventory/*CLUSTER_NAME*/hosts.yml python3 contrib/inventory_builder/inventory.py ${IPS[@]}
  • Change the value of kube_apiserver_ip in inventory/*CLUSTER_NAME*/group_vars/k8s-cluster/k8s-cluster.yml to IP of your preferred master node
  • Change the installation method of libselinux-python in roles/bootstrap-os/tasks/bootstrap-centos.yml From ``` package: name: libselinux-python state: present
to

raw: yum install libselinux-python * Enable the `local_volume_provisioner_enabled` field in `inventory/pre-prod/group_vars/k8s-cluster/addons.yml` : local_volume_provisioner_enabled: true local_volume_provisioner_namespace: kube-system local_volume_provisioner_storage_classes: local-storage: host_dir: /mnt/k8s-volumes mount_dir: /mnt/volumes * Change the IP in `inventory/pre-prod/group_vars/k8s-cluster/k8s-cluster.yml` file and enable persistent volumes: kube_apiserver_ip: "IPS_OF_MACHINES" # "{{ kube_service_addresses|ipaddr('net')|ipaddr(1)|ipaddr('address') }}" persistent_volumes_enabled: true `` -ansible-playbook -i inventory/pre-prod/hosts.yml --become --become-user=root -u root cluster.yml`

  • Copy /etc/kubernetes/admin.conf from the cluster node to your host one and configure kubectl to use use it by running export KUBECONFIG=*path_to_your_config*

Further infrastructure provisioning

Connect to your Kubernetes cluster and run perform the following steps:

  • Run helm init --force-upgrade
  • Provision a MySQL database, either by running it in the cluster or in a dedicated instance(recommended)
  • Provision an object storage server/bucket either by using any S3-compliant service or by deploying (Minio)[https://github.com/minio/minio] into the cluster
  • Prepare the deployment values for an Ingress Controller to load-balance all the incoming traffic to the cluster: curl -o ingress.yml https://raw.githubusercontent.com/helm/charts/master/stable/nginx-ingress/values.yaml
  • And update the config file with the following values:
<   hostNetwork: false
---
>   hostNetwork: true
<   dnsPolicy: ClusterFirst
---
>   dnsPolicy: ClusterFirstWithHostNet
<   reportNodeInternalIp: false
---
>   reportNodeInternalIp: true
<     useHostPort: false
---
>     useHostPort: true
<       http: 80
<       https: 443
---
>       http: 30001
>       https: 30002
<     type: LoadBalancer
---
>     # type: LoadBalancer
<     # type: NodePort
<     # nodePorts:
<     #   http: 32080
<     #   https: 32443
<     #   tcp:
<     #     8080: 32808
---
>     type: NodePort
<       http: ""
<       https: ""
<       tcp: {}
<       udp: {}
---
>       http: 32080
>       https: 32443
  • Deploy the Ingress Controller using specified values helm install --name ingress stable/nginx-ingress -f ingress.yml

Congratulations, your infrastructure is now ready for the deployment!

Vault configuration

To configure the Vault backend(e.g. OVH Openstack Swift), edit config/environments/*env*/vault.yml:

    storage:
      swift:
        auth_url: "https://auth.cloud.ovh.net/v2.0"
        container: "CONTAINER_NAME"
        username: changeme
        password: changeme
        tenant: "changeme"
        region: "REGION_NAME" # Should be uppercase
        tenant_id: "changeme"

You can find more information on Openstack Swift Vault backend here

Database configuration

Barong database settings can be configured in config/environments/*env*/barong.yml:

 db:
   name: changeme
   user: changeme
   password: changeme
   host: changeme # SQL hostname(e.g. 42.1.33.7)
   port: "changeme" # Usually 3306

Google Container Registry Access

To access GCR, create a service account with correct access rights and add a pull secret to the cluster:

kubectl create secret docker-registry pull-gcr --docker-server=https://gcr.io --docker-username=_json_key --docker-email=user@example.com --docker-password="$(cat *PATH_TO_JSON_FILE*)" -n *deployment_id*-app

Add the secret name to the configuration of any component that needs to use a private image pullSecret: pull-gcr.