Kubernetes NGINX Load Balancer Setup Guide

Introduction

This guide walks you through setting up an NGINX load balancer for Kubernetes using Docker Compose. Ensure all pre-requisites are met before proceeding.

Pre-Requisites

Step 1: Create NGINX Configuration

Create the NGINX configuration file to define load balancing rules:

nano /mnt/k8s/conf/nginx.conf

Add the following content:

server {
  listen 443 ssl;
  server_name your.domain.com;

  ssl_certificate /etc/letsencrypt/live/your.domain.com/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/your.domain.com/privkey.pem;
  ssl_dhparam /etc/nginx/ssl/dhparam.pem;
  include /etc/nginx/options/options-ssl-nginx.conf;

  location / {
    proxy_pass https://kubernetes_api;
    proxy_ssl_verify on;
    proxy_ssl_trusted_certificate /etc/nginx/ssl/ca.crt; #Make this available to NGINX
    proxy_ssl_name kubecontrol; #Kubernetes Subject Alt Name - see CA certificate
    proxy_ssl_server_name on;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
  }
}
upstream kubernetes_api {
  least_conn;
  server 192.168.2.52:6443 max_fails=3 fail_timeout=5s; #You kubernetes control node IP
}

To retrieve your proxy_ssl_server_name, you can run the following command your control node:

openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text -noout | grep -A 1 "Subject Alternative Name"

Make your Kubernetes CA certificate available to NGINX:

scp [email protected]:/etc/kubernetes/pki/ca.crt /mnt/k8s/ssl/ca.crt

Step 2: Create Docker Compose File

Set up a Docker Compose configuration for the NGINX load balancer:

nano ~/docker-compose.yml

Add the following content:

services:
  nginx:
    image: nginx:latest
    container_name: nginx-k8s-lb
    ports:
      - "443:443"
    volumes:
      - /mnt/k8s/certs:/etc/letsencrypt
      - /mnt/k8s/ssl:/etc/nginx/ssl
      - /mnt/k8s/options:/etc/nginx/options
      - /mnt/k8s/conf:/etc/nginx/conf.d
      - /mnt/k8s/logs:/var/log/nginx
    restart: always

Step 3: Start the Load Balancer

Use Docker Compose to start the NGINX load balancer:

docker-compose up -d

Step 4: Test and verify

Test that NGINX is running correctly and serving traffic:

docker exec nginx-k8s-lb nginx -t
docker exec nginx-k8s-lb nginx -s reload
curl -k https://your.domain.com

Step 5: Create service account

On your control node create a service account to use against your new load balancer:

kubectl create serviceaccount admin-user -n kube-system

Step 6: Bind the account to the cluster administrator

On your control node bind the account to the cluster administration role (or which ever role you decide):

kubectl create clusterrolebinding admin-user-binding \
--clusterrole=cluster-admin \
--serviceaccount=kube-system:admin-user

Step 7: Create an access token for your new service account:

On your control node create a token to be used to query your Kubernetes API through the load balancer:

nano admin-user-token.yml
apiVersion: v1
kind: Secret
metadata:
  name: admin-user-token
  namespace: kube-system
  annotations:
    kubernetes.io/service-account.name: admin-user
type: kubernetes.io/service-account-token

On your control node apply the configuration:

kubectl apply -f admin-user-token.yaml

On your control node extract the token in base64:

kubectl get secret admin-user-token -n kube-system -o jsonpath='{.data.token}' | base64 --decode

Step 8: Create a kubeconfig to query your cluster through the load balancer:

On a machine with kubectl installed, create a config:

nano ~/admin-user.kubeconfig
clusters:
- cluster:
    certificate-authority-data: ca.crt base64 encoded
    server: https://your.domain.com
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: user
  name: user@kubernetes
current-context: user@kubernetes
kind: Config
preferences: {}
users:
- name: admin-user
  user:
    token: token

Make sure to replace the above your configuration.

Step 9: Use and test your new configuration

On the machine which houses your new kubeconfig, test connectivity:

kubectl --kubeconfig=~/admin-user.kubeconfig get nodes

Test directly from your load balancer container:

docker exec -it nginx-k8s-lb curl --cacert /etc/nginx/ssl/ca.crt \
-H "Authorization: Bearer token \
https://192.168.2.52:6443/api/v1/nodes

Step 10: Troubleshooting

Use the following commands for troubleshooting:

docker exec nginx-k8s-lb nginx -t
docker exec nginx-k8s-lb nginx -s reload
docker exec -it nginx-k8s-lb curl --cacert /etc/nginx/ssl/ca.crt https://192.168.2.52:6443/version