Innovation / Solutions / Software / Cloud


AgileGuru Engineering blog on innovative solutions and technical excellence by engineers and architects.

Setting up a simple disaster recovery (DR) mechanism for Jenkins running in Google Kubernetes Engine (GKE) using Google Cloud Storage (GCS) buckets and FUSE persistent volumes.

Optimized DR Mechanism For Jenkins Using GCS Bucket in GKE

Guru Raghupathy, 01 January 2025

In today's ever-evolving digital landscape, customer expectations are very unforgiving. Matching these expectations and being always at the forefront is paramount for any successful business. Applications and Services including DevSecOps tools should always be available to make this happen. In this post we will provide an easy to implement solution for DR / High availability Mechanism for Jenkins running in Google Kubernetes Engine (GKE) using Google Cloud Storage (GCS) buckets with FUSE persistent volumes.

Assumptions

  • Jenkins in running as a deployment workload in GKE.
  • You can install plugins in Jenkins.
  • You can create a K8S Service Account and IAM account in GKE / GCP.
  • You have enabled FUSE CSI driver in GKE.
  • You have port 8080 available for fuse metric monitoring in the pod.
  • You can change the default 8080 port of jenkins to something else.

Proposed Architechture



Responsive image

Implementation Guide



1. Terraform Module To Create Bucket :
Responsive image

2. Create Bucket and Service Account in GKE with Workload Identity :
Responsive image

3. k8s Role Binding for the Service Account :
Responsive image

4. k8s Fuse PV and PVC :
Responsive image

5. k8s Jenkins Application Deployment with annotation gke-gcsfuse/volume :
Responsive image

6. Jenkins Install Periodic Backup Plugin :
Responsive image

7. Jenkins Listing Periodic Backup Plugin in Jenkins Management :
Responsive image

8. Jenkins Restoring Option Periodic Backup Plugin :
Responsive image

9. GCP Cloud Storage Bucket Listing matching Jenkins Data Backup :
Responsive image

10. GKE Pod having FUSE ( GCS ) Disk at /jenkins_backup with 1 PB capacity :
Responsive image

Conclusion

This DR solution provides a robust, cloud-native approach to backing up and recovering Jenkins in a Kubernetes environment. Whilst this solution does use GCS for storage, it offers a practical balance between complexity and effectiveness for most organizations. Remember to regularly test your DR process and keep documentation updated to ensure it remains effective when needed. Its main strengths are :

  1. Simplicity : Uses native GCP services (GCS, GKE) with straightforward configuration
  2. Cost-Effective : Leverages GCS for storage, which is cheaper than persistent disks
  3. Reliable : Automated backup process with built-in capabilities
  4. Secure : Implements modern security practices with Workload Identity

Author : Guru Raghupathy , 01 January 2025