There is a better way to Kustomize Kubernetes deployments for multiple environments by using a centralized parameter and secret store with templating.
To follow along with this blog post, you should have the following:
- A CloudTruth account. If you don’t have one, you can sign up for free here.
- A code editor, like VS Code
What is Kustomize?
In many organizations, you’ll see multiple environments, sometimes including:
- Staging (or UAT)
Each of those environments has configurations that need to be created, managed, and maintained. It can be a hassle as it’s more deployment configurations, parameters, config data, and code to manage.
Kustomize gives the ability to segregate environments (like dev, staging, and prod) and have configuration files that are specific to each environment. The configuration files are called
kustomization.yaml. Then, the Kustomize configuration files point to a “base” configuration, which is essentially a bunch of Kubernetes Manifests. Kustomize then changes the “base” configuration of the Kubernetes Manifests with the values from the Kustomize configuration files.
Below are two screenshots for a visual representation.
How To Use Kustomize
To use Kustomize, you need three things:
- A folder hierarchy (like in the screenshot above)
- Kubernetes Manifest(s)
- One or multiple (depending on how many environments you have)
First, create three directories called
Next, inside of the
base directory, create two files:
Inside of the
deployment.yaml, add the following Kubernetes Manifest:
apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment spec: selector: matchLabels: app: nginxdeployment replicas: 2 template: metadata: labels: app: nginxdeployment spec: containers: - name: nginxdeployment image: nginx:latest ports: - containerPort: 80
Inside of the
kustomization.yaml, add the following Kustomize configuration file:
apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - deployment.yaml
You’ve created two things; the Kubernetes Manifest that Kustomize will look for and the Kustomize config file that tells Kustomize what Kubernetes Manifests you want to use.
Next, inside of the
dev directory, create a new file called
kustomization.yaml and add the following code:
apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - ../../base/ replicas: - name: nginx-deployment count: 1
What the above
kustomization.yaml file is doing is pointing to the
base directory, which is where the Kuberentes Manifest and Kustomize config base files live. Then, it’s specifying in the configuration that for the
nginx-deployment app, only add one replica.
Now, you can run
kustomize build and see the output, specifying only one replica.
How about Configuration Data?
Here’s what Kustomize doesn’t solve:
- You still have multiple environment/parameter/config data files
- You have to have multiple directory structures for different stages (dev, staging, prod, etc.)
- There’s no central way to pass in config data/parameters from one centralized location
- Kustomize focuses on replacing values instead of managing values for each environment/stage
Although the above is great and Kustomize is an amazing tool, you still need a way to centrally manage the config data that you’re passing in. You can use both Kustomize and CloudTruth to segregate environments, config files, and manage multiple parameters.
Taking a look at the screenshot below, you can see that:
- There’s a project called
- There’s a parameter called
replicaCountthat has a value of 1 Kubernetes replica
This means that using CloudTruth, Kustomize will look at CloudTruth to retrieve the
replicaCountvalue for the k8s manifest.
Inside of the
kustomization.yaml file, you’d specify the CloudTruth parameter name
replicaCount in two curly brackets, as shown in the code snippet below:
apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - ../../base/ replicas: - name: nginx-deployment count:
The last step is to run the CloudTruth command below to:
- Specify what project you want to pull a parameter from
- The environment
- Specify the command you want to run. In this case, it’s the
cloudtruth --project "kustomize" --env development run -- kustomize build
The output would be a Kubernetes Manifest that contains the 1 replica.
In the DevOps world today, there are a million tools to choose from. Because of that, you may sometimes see that tools are chosen that can do a job, but may not be the right fit. However, combined with other tools and platforms, they become a perfect match.
If you’d like to get started with CloudTruth today, there is a free forever version here.