Guru Raghupathy, 15 January 2025

Managing GKE (k8s) clusters and applications inside them has become a never ending battle for many of us. Managing various attributes like node-pools / add-ons / ingress controllers / SSL certificate manager / rollout of applications and its associated configuration accurately has become cumbersome for many. This has become more common given the rise of micro-services and event-driven architecture having many components.

Assumptions

• You deploy your application in a GCP platform GKE cluster.
• You manage your infrasture using Terraform (IAC) Infrastructure as code tool.
• You want to manage your apps in a seamless directory hierarchy.

1. Automating management of GKE ( K8S ) clusters

GKE / K8S cluster setup is a complicated process. A good approach is to automate the creation part as a IAC artefact. We will be using terraform as the IAC tool in this article. Creating modules with parameters / attributes for every resources is the key to maintain environment consistency (parity). We will have 3 modules in our repo as provided at https://github.com/agileguru/gke_nginx_kcert_quick_start. 1. GKE Cluster module having two node pools. 2. nginx ingress controller module using helm chart provisioner. 3. kcert letsencrypt SSL certificate provider module for your public endpoints.

1.1 Provisioning GKE Cluster with multiple node-pools

In this module we have main.tf, variables.tf and outputs.tf for provisioning / parameterizing / resuable metadata for a GKE cluster respectively.

1.2 Installation of nginx ingress controller

In this module we have main.tf, variables.tf and an optional (empty) outputs.tf for provisioning / parameterizing nignx controller respectively.

1.3 Installation of kcert Letsencrypt SSL manager

In this module we we have main.tf, variables.tf and an optional (empty) outputs.tf for provisioning / parameterizing kcert SSL controller respectively.

1.4 Provision the cluster and controllers with terraform

• Change the project / region / zone name in variables.tf
• Change the bucket name in backend.tf after creating in gcp console
• Execute the following commands

terraform init
terraform plan -var-file=sample.tfvars ( change sample.tfvars if needed)
terraform apply -var-file=sample.tfvars ( change sample.tfvars if needed)

* After executing terraform commands, you will get the IP address of the LoadBalancer for domain registration.

* You can get the kubectl config usig the command …

gcloud container clusters get-credentials <cluster name> — zone <cluster zone>  — project <project id having the cluster>

2. Managing deployments using Kustomize

Cluster is now ready for workload deployment. We will be using Kustomize plugin to manage it in a easier way. Kustomize provides a solution for customizing Kubernetes resource configuration free from templates and DSLs. Kustomize lets you customize raw, template-free YAML files for multiple purposes, leaving the original YAML untouched and usable as is. Kustomize targets kubernetes, it understands and can patch kubernetes style API objects. We will use a simple use case for this article given below...

• We have 2 apps api-1 and api-2 based on tutum/hello-world image.
• We also have 2 k8s namespaces corresponding to DEV & SIT environments.
• We need to deploy the service and expose it via https (ssl ) with respective configurations / deployment / service and ingress mapping.
• We need to store this in a repository. For Demo it is stored in our repository at https://github.com/agileguru/kustomize_quickstart_demo

2.1 Creating the folder structure

2.2 Customise each environment using overlays

2.3 Change Ingress Host Name Mapping : Change hostname in *.json to look similar to below...

[
    {
        "op": "replace",
        "path": "/spec/rules/0/host",
        "value": "dev.agileguru.org"
    },
    {
        "op": "replace",
        "path": "/spec/tls/0/hosts/0",
        "value": "dev.agileguru.org"
    }
]

[
    {
        "op": "replace",
        "path": "/spec/rules/0/host",
        "value": "sit.agileguru.org"
    },
    {
        "op": "replace",
        "path": "/spec/tls/0/hosts/0",
        "value": "sit.agileguru.org"
    }
]

2.4 Deploying your Applications with kubectl apply -k

  $ kubectl apply -k overlays/dev
    namespace/dev created
    configmap/config-map-api-1 created
    configmap/config-map-api-2 created
    service/api-1-service created
    service/api-2-service created
    deployment.apps/api-1-deployment created
    deployment.apps/api-2-deployment created

2.5 Un-deploying your Applications with kubectl delete -k

  $ kubectl delete -k overlays/dev
    namespace "dev" deleted
    configmap "config-map-api-1" deleted
    configmap "config-map-api-2" deleted
    service "api-1-service" deleted
    service "api-2-service" deleted
    deployment.apps "api-1-deployment" deleted
    deployment.apps "api-2-deployment" deleted
    ingress.networking.k8s.io "app-ingress" deleted

Kustomize Best Practices

1. Things To You Need To Do

• Keep the Replica Count to 0 in the base configuration
• Always Specify the Namespace in Overlay kustomization.yaml
• Always Dry Run with yaml output to ensure accuracy
• Good Naming Conventions for folders and manifest files
• Keep Ingress Mapping in its own-folder
• Always have override.yaml in for each component

2. Things You Should Not Do

• Hard-Coding namespace in Base Configurations
• Mix Configurations and Application Code In the Same Folder
• Git branches for Environment configuration ( know as parity drift )

Related Resources

1. GKE Nginx Kcert getting Started Repo : https://github.com/agileguru/gke_nginx_kcert_quick_start
2. Kustomize Quick Started Repo : https://github.com/agileguru/kustomize_quickstart_demo
3. Kustomize Docs : https://kustomize.io/
4. Kcert GitHub : https://github.com/nabsul/kcert
5. Nginx Controller : https://kubernetes.github.io/ingress-nginx/
6. LetsEncrypt : https://letsencrypt.org/

Conclusion

After completing the above steps we have 1. k8s which is easy to manage / upgrade with nginx and kcert SSL certificate manager without having to manage you SSL certificates ever for your public endpoints. 2. Mechanism / Framework to manage your secure web based endpoints following IAC / Devops / DRY principles.

Author : Guru Raghupathy , 15 January 2025