Introduction
This guide walks you through setting up an NGINX load balancer for Kubernetes using Docker Compose. Ensure all pre-requisites are met before proceeding.
Pre-Requisites
- A functional Kubernetes cluster.
- A separate server or VM for the load balancer.
- Docker and Docker Compose installed on the load balancer server.
- You have obtained a certificate and accompanying private key.
- A DNS record pointing from your domain name to NGINX server IP.
- In my example below, a control plane node was at 192.168.2.52.
- You will notice /mnt/k8s/... throughout the guide. This is central storage location I use for my setup. Please change these locations to meet your needs.
Step 1: Create NGINX Configuration
Create the NGINX configuration file to define load balancing rules:
nano /mnt/k8s/conf/nginx.conf
Add the following content:
server {
listen 443 ssl;
server_name your.domain.com;
ssl_certificate /etc/letsencrypt/live/your.domain.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/your.domain.com/privkey.pem;
ssl_dhparam /etc/nginx/ssl/dhparam.pem;
include /etc/nginx/options/options-ssl-nginx.conf;
location / {
proxy_pass https://kubernetes_api;
proxy_ssl_verify on;
proxy_ssl_trusted_certificate /etc/nginx/ssl/ca.crt; #Make this available to NGINX
proxy_ssl_name kubecontrol; #Kubernetes Subject Alt Name - see CA certificate
proxy_ssl_server_name on;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
}
upstream kubernetes_api {
least_conn;
server 192.168.2.52:6443 max_fails=3 fail_timeout=5s; #You kubernetes control node IP
}
To retrieve your proxy_ssl_server_name, you can run the following command your control node:
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text -noout | grep -A 1 "Subject Alternative Name"
Make your Kubernetes CA certificate available to NGINX:
Step 2: Create Docker Compose File
Set up a Docker Compose configuration for the NGINX load balancer:
nano ~/docker-compose.yml
Add the following content:
services:
nginx:
image: nginx:latest
container_name: nginx-k8s-lb
ports:
- "443:443"
volumes:
- /mnt/k8s/certs:/etc/letsencrypt
- /mnt/k8s/ssl:/etc/nginx/ssl
- /mnt/k8s/options:/etc/nginx/options
- /mnt/k8s/conf:/etc/nginx/conf.d
- /mnt/k8s/logs:/var/log/nginx
restart: always
Step 3: Start the Load Balancer
Use Docker Compose to start the NGINX load balancer:
docker-compose up -d
Step 4: Test and verify
Test that NGINX is running correctly and serving traffic:
docker exec nginx-k8s-lb nginx -t
docker exec nginx-k8s-lb nginx -s reload
curl -k https://your.domain.com
Step 5: Create service account
On your control node create a service account to use against your new load balancer:
kubectl create serviceaccount admin-user -n kube-system
Step 6: Bind the account to the cluster administrator
On your control node bind the account to the cluster administration role (or which ever role you decide):
kubectl create clusterrolebinding admin-user-binding \
--clusterrole=cluster-admin \
--serviceaccount=kube-system:admin-user
Step 7: Create an access token for your new service account:
On your control node create a token to be used to query your Kubernetes API through the load balancer:
nano admin-user-token.yml
apiVersion: v1
kind: Secret
metadata:
name: admin-user-token
namespace: kube-system
annotations:
kubernetes.io/service-account.name: admin-user
type: kubernetes.io/service-account-token
On your control node apply the configuration:
kubectl apply -f admin-user-token.yaml
On your control node extract the token in base64:
kubectl get secret admin-user-token -n kube-system -o jsonpath='{.data.token}' | base64 --decode
Step 8: Create a kubeconfig to query your cluster through the load balancer:
On a machine with kubectl installed, create a config:
nano ~/admin-user.kubeconfig
clusters:
- cluster:
certificate-authority-data: ca.crt base64 encoded
server: https://your.domain.com
name: kubernetes
contexts:
- context:
cluster: kubernetes
user: user
name: user@kubernetes
current-context: user@kubernetes
kind: Config
preferences: {}
users:
- name: admin-user
user:
token: token
Make sure to replace the above your configuration.
Step 9: Use and test your new configuration
On the machine which houses your new kubeconfig, test connectivity:
kubectl --kubeconfig=~/admin-user.kubeconfig get nodes
Test directly from your load balancer container:
docker exec -it nginx-k8s-lb curl --cacert /etc/nginx/ssl/ca.crt \
-H "Authorization: Bearer token \
https://192.168.2.52:6443/api/v1/nodes
Step 10: Troubleshooting
Use the following commands for troubleshooting:
docker exec nginx-k8s-lb nginx -t
docker exec nginx-k8s-lb nginx -s reload
docker exec -it nginx-k8s-lb curl --cacert /etc/nginx/ssl/ca.crt https://192.168.2.52:6443/version