Kubernetes and Replicated StatefulSet with Real World Application

Introduction

These days Cloud computing is growing very fast, reason behind is, automated application deployment, automated scale up/down of application and it’s hardware resource based upon workload, easy availability, resilient architecture and operation, respectively low cost in comparison to dedicated data centers.

The hardware resource and physical devices sharing in Cloud computing emerges from a technology called virtualization. In this virtualization, normally a single bare metal device contains a VMware or other virtualization packages that creates many virtual machines and at background share the same physical devices to serve each client of virtual machine.

The next level of virtualization is Containerization. Containers are nearly same as Virtual Machines (VMs) with some changes, they share the same operating system but create isolation for each container and treat as separate deployment of application. Since there is no own operating system for each container it is considered light weight and does not take time at boot up when containers are created. It just shares the same underlying OS kernel to perform commands but maintain isolation level. Just same as VMs, container has own file system, memory, CPU, process space that provide isolation with other containers.

Below diagram depicts the containers structure:

Figure 1: Depicting containerization

The containers are giving flexibility to the platform owner that many application/instances can be deployed into single hardware infrastructure and sharing physical resources.

Now, the next quest is ‘How to manage containers?’

The container platforms are also providing the commands and utilities to manage the containers. But it requires manual intervention and, consider the case if some application is being deployed with hundreds of instances to justify the workload it will be very complex to monitor container state and manage the container manually.

The popular choice answer is ‘Kubernetes’.

Kubernetes is fastest growing powerful, extensible, and open-source platform which provides features to manage Container Runtime Interface (CRI) based containers. It supports both declarative with configuration and automated container management. In simple words, Kubernetes is a platform which maintains the state of containers as specified by administrator in form of declarative instruction or automated instruction.

Below diagram depicts the Kubernetes structure:

Figure 2: Kubernetes architecture

The Kubernetes manages the components called Pods, a Pod is the smallest component which is managed by the Kubernetes control plane and Kubernetes always maintain the Pods’ state as configured into Kubernetes by the administrator.

There are two types of Pods: Single Container Pod and Multi-container Pods

In Kubernetes there are two type of deployments in respect of Pods persistency:

- Stateless Pods: Those pods which are created by Deployment and it do not maintain the persistent state of pods.

- Stateful Pods: When we need to store some data or files along with Pods, it requires stateful pod because in case of pod restart the files or data should remain intact and application can start from there only. Especially, consider the example of databases which always stores the data into files.

Now, there is question with Stateful pods, when pods are automatically being managed and scaled up or down, how the files are maintained with each instance of application running.

This brings a concept called replication, replication is performed at the time scaling up or down to maintain the data with instance of application running. There is a component called StatefulSet is created for stateful deployment.

About the Document

This document created to demonstrate the applicability and implementation of StatefulSet with replication into Kubernetes along with Persistent Volume and Persistent Volume Claim.

Introduction of Technologies Used

Docker — Docker is a popular container platform based upon Container Runtime Interface (CRI), allows to management of application image and running images to create containers. It internally uses containerd.

Registry — Registry, a place where created images are stored and when required it is fetched from the registry path.

Kubernetes — A widely used container orchestrator platform which supports manually as well as automated container (containers which are implementing CRI) management.

MySQL — Widely used community edition database which support latest SQL specification.

Java — A platform independent and most popular application programming language widely used for web and enterprises application development. It has been used here to write code for microservice.

Spring Boot — Popular Java based, RAD category platform helps us in creating embedded server based microservice using tomcat using Dependency Injection (DI) and Inversion of Control (IoC).

Maven — Java based build tool, it helps in creating Java based projects and package then as deliverables, including development cycles e.g. resolve dependency, compile, package, unit test, installing into repository and deployment.

Docker Spotify — A maven plugin which interacts with docker and create application image, tag, and push into registry.

Percona Xtrabackup — The mysql database does not support database replication natively from the application side. So, to implement replication, a third part open source component called Xtrabackup is used.

Target Audience

The discussion here is around the Kubernetes and its StatefulSet deployment. Therefore, the audience must be aware of below terminologies and technologies:

• Basics of Cloud Infrastructure
• Docker, registry, compose, volume
• Kubernetes, Pods, ConfigMap, Services
• Microservice creation with Spring boot and Maven build tool

The defined target audience are:

• Microservice developers (by whom application needs to be deployed on Cloud)
• Technical Architects

Abbreviations

Table 1: Abbreviations

Business Needs

It is very common need to have persistent data along with application. When application is being designed, it most of the cases it is required to persists data into some permanent storage and retrieval of the same data whenever it is required.

The proven approach is to store data effectively is database, typically application uses some database being the application to store the data values for the application and it is being retrieved by application and again changes stored into database. The database typically stores the data into flat files stored into the filesystem and supports.

Now, if we deploy application on containers, the files will be created into the container’s local filesystem (just taken above example). Consider automated container management scenario where containers are being created or delete at runtime based upon workload. Normally, with container creation, new set of files created and when containers are deleted (because of any reason) the files will destroy with containers.

So, the first need is to use a persistent volume along with containers to persists the data permanently irrespective of containers state.

Now, coming to second business need, when containers are managed by Kubernetes, understand the case:

Suppose one mysql database has been deployed into StatefulSet deployment with some volume mapped with it. When workload increase, there is need to increase the instances of application which replication controller automatically increase the application instance, but they all connect to single volume (single set of files) that increase the I/O operation on the volume but there is always a limitation of I/O operation with any disk so as result, at a point of time, increasing instance of mysql will not suffice.

In this case, solution is to replicate the mapped volume as well and create multiple instances so that workload can be distributed as well as I/O operation can also be distributed. Also, in case of DR setup, replication of volume is required when it switches to DR instances running on another node.

We are going to discuss the implementation of replicated StatefulSet to overcome from these situations.

Problem Statement

In this document, we are going to discuss the applicability and implementation of replicated StatefulSet with Kubernetes. For this purpose, a use case of common banking solution has been taken.

In case of Kubernetes deployments, where data values for the application to be stored to run business functionality of application ‘what will be approach?’.

The above line describing about the problem statement in a single line. To make descriptive we have created a use case for the same:

Create a core banking application which have two microservices:

- Customer Management Service (CSM) — This microservice expose the endpoints for customer, account and bank branch static data management which will be utilized during the banking transaction.

- Core Banking Transaction Service (TRN) — It provides basic banking functionalities like deposit by cash, withdrawal by cash, fund transfer, transaction enquiry, statement etc.

Both microservice will be deployed with two replicas at minimum to cover the required workload. Both microservice will use the mysql database as backend database to store the values and too deployed on the Kubernetes. The database files should be persistent and when database replica is increased, the persistent files will also be replicated along with the mysql.

The problem statement is, to demonstrate the Kubernetes capabilities on deployment of stateful application using StatefulSet, availability of application using replication controller, storage management using persistent volume and persistent volume claim with industry-based example.

Mysql does not support replication natively, so when it is being implemented into the application with StatefulSet, the replication will create a challenge. To overcome from the database replication, a popular component Percona Xtrabackup is used as solution.

When we talk about the any deployment where Kubernetes manages the deployment, the container technology automatically comes in. Thus, setting up docker, creating application image, setting up registry and command for underlaying operating system.

Note: This demonstration does not cover automation, automated testing, DevOps, and other concerns. It focuses on the scope defined for the assignment. However, in later stages can be expanded as provisions are already in place.

Solution with StatefulSet Implementation

The solution for the said problem statement is to create application and deploy it on Kubernetes platform.

Create two applications’ microservices as described into the Problem Statement section by using Spring boot 2.x and create microservice runnable jars by following the design described here,

Figure 3: Proposed application architecture

Description of above diagram:

There will be two microservices created by Spring boot. Both will be using mysql as backend database to store data

There will be single mysql server hosting both database for the microservice

Both microservice will expose endpoints using Swagger UI for the end user

When transaction will be performed, the transaction microservice makes internal call to customer management service microservice

Since mysql does not support database replication from its native database platform, it is required to implement some component to replicate the database. There is open source component from Percona called Xtrabackup, is very popular solution for database replication.

The Xtrabackup will also be deployed as part of mysql deployment which helps into the replication of database.

Once application microservice will be created, the next step is to design the deployment diagram with respect to Kubernetes:

Figure 4: Depicting application deployment and containers into Kubernetes platform

Description of diagram:

• There will be a Kubernetes cluster
• It must have one master and worker node as per typical installation of Kubernetes
• A namespace to be created to segregate the workspace for ‘bank-apps’
• To showcase replicated SatefulSet, mysql:5.7 (Innodb) will be used
• - The StatefulSet name will be mysql
• - There will be two replicas so by adding ordinal index Pods name will be like mysql-0 and mysql-1
• - The mysql-0 will be the master and mysql-1 will be replicated database
• - Each Pods will have two containers; mysql container will run mysql:5.7 server and xtrabackup container will run to replicate master database log file to replicated instance of database
• - xtrabackup is a popular Percona open source application widely used for mysql data replication
• - There will be two set of StorageClass, Persistent Volume and Persistent Volume Claim to serve the dynamic volume need of each Pod running for mysql StatefulSet
• - When replica is scaled up, the needed dynamic volume is created by the Kubernetes, need to create Persistent volume already created in case if we want to manage PV manually
• - Each instance of mysql:5.7 server will run two databases named: project_csm to serve CSM Microservice and project_trn to serve TRN Microservice
• - The StatefulSet will be exposed by Headless service which will be accessed by application microservice by name as mysql-0.mysql:3306
• There will two application microservices will run into Stateless single container Pod and replica can be increased by need or load.
• Each microservice will access master database pod by the pod name and container name as mysql-0.mysql:3306 and access corresponding database like project_csm and project_trn

The default network will be used as of now that’s why network controller has not been depicted here.

As load increase or based upon the deployment configuration, the PODs will be scaled up or down automatically by the Kubernetes.

Now, when we implement the above described solution, it will create the required persistent volume, persistent volume class and StatefulSet to deploy the mysql database.

Technical Components used (with their declarative configuration in form of yaml) here,

Storage Class

A storage class resource defines the quality of service level, backup policies and other arbitrary polices for the storage volume which uses the created storage class.

Simply, it will require to create a declarative configuration as,

Code Snippet 1: Storage Class

There are number of provisioner available for each Cloud platform (e.g. AzureDisk, AzureFile, AWSElasticBlock ScaleIO etc.), which need to be used based upon their service level and applicability to the project.

To create storage class resource, just run below command:

kubectl create -f storage-class-0.yaml

Persistent Volume

It is just a storage which will be used into Kubernetes Cluster provisioned by manually by administrator or dynamically provisioned by specified Storage Class.

To create persistent volume, we just have some volume and provisioner and need to create a configuration yaml like:

Code Snippet 2: Persistent Volume

Here,

StorageClass defines the provisioner and policy associated with Storage class

Capacity describes the capacity of storage

Access mode defines the applicability of pods to read or write

The host path is based upon the type of persistent volume. There are number of persistent volume available e.g. AzureDisk, AzureFile, FlexVolumes, AWSElasticBlockStore etc.

Note: Here, /host_mnt/ represents the local host binding for shared path of Windows operating system.

To create persistent volume, just run below command:

kubectl create -f persistent-volume-0.yaml

Persistent Volume Claim

It is just request for storage by the user or application and will consume the persistent volume sources when request made.

The request will consume specified amount of storage with defined access mode from the available persistent volume. It is mainly used with StatefulSets to dynamically allocate the persistent volume to the replicas.

To create persistent volume resource, need to create a yaml configuration as,

Code Snippet 3: Persistent Volume Claim

When request come from the StatefulSet with specified parameters, it allows to consume the persistent volumes available with same parameters e.g. storage class, access modes and size of storage needed.

To create persistent volume claim resource, just run below command:

kubectl create -f persistent-volume-claim-0. yaml

Stateful Deployment

A Stateful deployment create with StatefulSet component into the Kubernetes. Once it is created, it starts creating of Pods and its containers based upon the definition of StatefulSet deployment.

The persistent volume claim defined into the StatefulSet requests the Persistent Volume needed for the pod instance going to start.

The replication controller of Kubernetes automatically manages the Pods count based upon the configuration.

As solution, a StatefulSet designed with two init and two containers as,

Init Containers:

1. It will run when pod initialization started. It will generate server id from the pods’ ordinal index, adds generated server id to server-id.cnf and copy the configurations from the config map.

Code Snippet 4: StatefulSet Init Container 1

2. The second init container will also run at initialization of pod and ensure the copying data from the previous running mysql server instance. It first checks, the data is already there then exit or ordinal index is 0 then exit.

Code Snippet 5: StatefulSet Init Container 2

Containers:

1. Mysql container to start on default port 3306

Code Snippet 6: StatefulSet Container 1

2. Percona Xtrabackup to get back up from the previous instance running, if it is running the master (means 0 ordinal index it will just run the Xtrabackup and expose the 3307 port for getting data to other replicas.)

Code Snippet 7: StatefulSet Container 2

Service

A service is resource which defines the access policy for the pods.

It simply contains the type of service definition and based upon type the applicable expose parameters

To create service into Kubernetes simply create the configuration which selects pods and expose as configured into yaml:

Code Snippet 8: Kubernetes Service

ConfigMap

A config map in Kubernetes is way to define the unencrypted key and value pair.

It can be created by yaml configuration e.g.

Code Snippet 9: Kubernetes Config Map

Application deployment

In Kubernetes, deployment is performed by declarative yaml configuration. Once all configuration yaml created only need to run the yaml with Kubectl command and it will create the required resources.

The next stage is to verify deployment, to do this, kubectl provides several commands to verify the deployment. Also, Kubernetes dashboard can be deployment to see the Kubernetes resource from the web page.

Suggested Approach

When application will be deployed with Kubernetes, it is suggested to follow the microservice design pattern where a complex application is broken down into smaller technical components and each component will have its’ dedicated mechanism from development to deployment.

Once, all microservices will be deployed and working, they collectively serve the intended complex flow of application. It brings easy development and maintenance, managing of application in Cloud platform with effectiveness, can provide processing power where exactly it is required and so on (refer the microservice design pattern by Martin Fowler and Chris Richardson from reference section).

The database replication should follow the replication pattern defined for databases, here, one master has been created and there will be many replicas for the same. Whatever DML will executed into master will be replicated to other instance and query can be performed any of replica (refer the replication design pattern by Bruce J Mack from the reference section).

The solution described in solution section has been designed for a typical use case taken to showcase the applicability of replicated StatefulSet with Kubernetes with Docker environment, it follows the microservice design pattern and replication design pattern for the database replication. This use case can be used into various situations with required modification but the overall solution for using the persistent volume and persistent volume claim will remain same.

While designing the solution for any Kubernetes deployment, it should be kept into consideration that when StatefulSet are created, it will need the Kubernetes replication controller to manage in special cases to maintain persistent volume and persistent volume claim with replicated instance. Also, replication of persistent volume must be performed based upon need as here with use case xtrabackup has been used to replicate the logfiles of database to each persistent volume.

However, it is always required to evaluate in microservice pattern about the applicability of persistent volume and StatefulSet and its requirement. Because stateless pods replication is quite faster and easy. But whenever, it comes to situation where data files or database deployment happen, it always should be in StatefulSet deployment.

There need to be take care of persistent volume declarative parameters very carefully, especially on reclaim policy which define persistent ability of volume data after work completed like retain, delete, recycle. Also, volume mode and expansion policy are quite important.

Reclaim Policies, e.g.

Retain, it retains the persistent volume even if persistent volume claim is deleted. It requires manual deletion of persistent volume to reclaim.

Delete, the second reclaim policy is delete, it removes the persistent volume as well as associated storage asset when persistent volume claim removed.

Recycle, per perform basic cleanup of persistent volume when persistent volume claim is removed. Additionally, recycle reclaim policy has been deprecated by Kubernetes recommends using dynamic provisioning.

There should be some command or example YAML file details to handle these.

Code Snippet 10: Persistent Volume’s Storage Class retention policy and volume expansion

Normally, the StatefulSet are created for database is created with headless service and is always accessed by the service name instead of exposing the service with specific ip. The reason behind it is, when StatefulSet deployments are access by name, it automatically routed to the different instances running for the StatefulSet instances based upon load and other parameters. Additionally, it makes application highly available, the available instance of StatefulSet starts responding if some instances are down, and replication controller automatically starts the pods to maintain the required set of replicas.

Code Snippet 11: Headless Service without IP

Along with replication of persistent volumes, it is recommended to take backups of the volumes on the specific intervals so that in case of any failure recovery can be performed and there will not be any data loss.

Conclusion

The StatefulSet in Kubernetes provides and state maintaining by the Pods along with scale up and down as required from the application deployment design. It also maintains the persistent volume consumption by using the persistent volume claim.

The application must be evaluated and the microservices and deployable components must be marked as stateful or stateless as per their requirement. Based upon the applicability the StatefulSet create.

Before creating StatefulSet, the persistent class, persistent volume and persistent volume claim must be created with needed parameters defining the policy of persistent volume like size, reclaim policy, expansion policy. Also, there must be provision volume space which can be consume when the replica will be increased for the StatefulSet.

The replication of persistent volume must always be into our design consideration because in different application use case it will require different solution based upon application need.

The solution designed here, is just a case where StatefulSet has been demonstrated with an application use case. The same solution can be implemented with some changes if required based upon the need for the application.

Links of Work on GitHub

There is a bank application created to demonstrate the entire solution and all artifacts has been placed into GitHub. Below are links to download:

• https://github.com/siddhivinayak-sk/k8s-projects/tree/main/k8s-csm-master
• https://github.com/siddhivinayak-sk/k8s-projects/tree/main/k8s-trn-master
• https://github.com/siddhivinayak-sk/k8s-projects/tree/main/kb-statefulset-sample-artifacts-master

References

• Microservice application design pattern by Chris Richardson author of microservice pattern — https://microservices.io/patterns/microservices.html and Martin Fowler https://martinfowler.com/articles/microservices.html
• Kubernetes official documentation and details — https://kubernetes.io/ and https://kubernetes.io/docs/home/
• Docker official website and documentation — https://www.docker.com/ and https://docs.docker.com/
• Pivotal Spring application development — https://spring.io/
• Apache maven the build tool — https://maven.apache.org/
• Spotify plugin build docker image — https://github.com/spotify/docker-maven-plugin
• Percona Xtrabackup — https://www.percona.com/doc/percona-xtrabackup/8.0/index.html
• Database replication by Bruce J Mack — http://www.brucejmack.net/guides/Pattern%20Documents/database/data_replication.htm

About the Author

Sandeep Kumar holds Master of Computer Application degree has been Java developer having 10 years of working experience. He has experience designing and development of enterprises applications into various education, content, laboratory, and banking domains, got various appreciation for providing solutions including spot appreciation for Glassfish to JBoss migration project. He secured Google Cloud Developer certificate and participated into OCI training. He is a part of HCL-ERS platform as Sr. Lead developer.