Integrating Terraform and Helm using helm_release
A blogpost about our experience integrating Terraform and Helm using the helm_release resource.
Omarm datagestuurd succes: meld u aan voor onze nieuwsbrief.
Abbonneer op onze nieuwsbrief en ontvang deskundige inzichten, bruikbare strategieën en verhalen uit de echte wereld die u zullen begeleiden naar het behalen van datagedreven succes.
An experience using Terraform and the
Terraform, Helm, and Kubernetes are all powerful tools for managing and deploying containerized applications on cloud environments. Terraform is an infrastructure-as-code (IaC) tool that allows you to create, manage, and version your cloud infrastructure. Helm, the package manager for Kubernetes, makes it easy to install, upgrade, and manage Kubernetes applications. And Kubernetes itself is an open-source container orchestration system that automates the deployment, scaling, and management of containerized applications. Combining these three tools allows you to automate the deployment and scaling of your applications on a Kubernetes cluster while maintaining the flexibility of your infrastructure.
In the last decade, IaC has made a profound impact on the industry. It is a practice that has proven itself, just like containers and Kubernetes, which are here to stay. Combining these practices in the form of Kubernetes, Terraform and Helm, can however be challenging. In this blog post, we will explore the different aspects of the Terraform helm_release module, and propose an alternative that might work better for your workflow.
An introduction of
The helm_release resource allows one to install a set of Kubernetes charts using the Helm tool, via a Terraform resource. What the resource does behind the scenes is using the
helm upgrade command to either install or upgrade your chart.
An example usage of the helm_release resource for a local chart is the following:
Such a chart could be installed with Terraform, with a Helm provider installed and access configured, for example with a kubectl configuration:
Terraform swallows all output
Conceptually this is great, as you can resort to just one tool to deploy and manage your infrastructure and applications. There are, however, several challenges using the helm_release resource.
When preparing for an upgrade, you naturally use the
terraform plan command to check the changes you have made. This is the first challenge when using helm_release, as we cannot see any diff or output in terms of changes for the Helm charts. The only output we ever get to see is:
As you can see in this example, the value of the Helm variable
worker.workerMemory has not changed, but still we got a diff indicating that it changed to
null and back to
(This is an altered example of an output from a real CI/CD pipeline where the number of variables is much larger than this. We get 200 line diffs every single time for things that have not changed. Note that the diff does not show up if nothing changed at all.)
When something goes wrong with Helm
If the combination of the Helm template and the Terraform variables it received created an error,
terraform apply will hang indefinitely. The command will give no output whatsoever on why the deployment failed.
A shortened example of this:
Once this happens you can use the
helm CLI tool to do some debugging, but this is definitely not the best experience.
When something needs to change
When something has to change in the Helm chart, it is not always the best experience deploying these changes. At times Terraform/Helm do not pick up the changes made in the Helm charts. The declarative approach of Terraform and Helm is a godsend, but if it does not work you will waste a lot of valuable time fixing the bug. One could try creating and destroying Helm charts at your behest, assuming that recreating them would solve these problems. However, two issues come up in this scenario:
- The Helm charts are managed by Terraform, if we make any changes outside of Terraform we are in for trouble, as that means Terraform and its state will get out-of-sync.
- Within Terraform we can directly alter (create/delete) specific Terraform resources. However, this also means that that is all we can do: we cannot change a particular aspect of a Helm chart. This is especially a problem if you have one parent Helm chart and subcharts, which is actually a great way to package your Helm charts.
An alternative approach to Helm & Terraform
Given the experiences described above, you might want to consider not managing your Helm charts with Terraform. However, all is not lost. A simple, yet effective approach to combining Helm & Terraform within one CI/CD pipeline does exist.
We could set this up as a two-step-approach. First, we make sure our Terraform changes have been made. Second, we avoid using
helm_release and use Helm natively to update our applications and Kubernetes infrastructure. This way we get the best out of both worlds, and can still natively interface and debug with our Terraform and Helm software. Moreover, as our Kubernetes applications might depend on resources that we have built in the Terraform step, we can send the variables that Helm needs in the second step.
Let’s look at an example of the second step:
This approach effectively means that we instead call the two tools separately, possibly linking them together using shell scripts.
In this snippet we upgrade our example chart (and install if needed) but can still have one place where we manage our variables: Terraform. As an example, we pass in the postgres connection string into our Helm chart, which might come from the managed postgres instance (from your cloud providers) you built earlier using Terraform code.
To integrate this into a GitHub action workflow, we could build a CI/CD pipeline for deployments:
This would perform
terraform plan everytime you push to GitHub, and perform a deployment when merging or pushing to the staging or main branch.
To conclude: Helm and Terraform
Terraform, Helm, and Kubernetes are very powerful technologies, however, integrating these in nice workflows to be employed in CI/CD pipelines can get complicated. Here we have presented some of our learnings on this topic. We hope that this helps others on their journey of integrating technologies and automating their software pipeline. Happy coding!
About the author
Maximilian is een liefhebber van machine learning, ervaren software-engineer en mede-oprichter van BiteStreams. In zijn vrije tijd luistert hij naar elektronische muziek en houdt hij zich bezig met fotografie en hiken.Meer lezen
Enjoyed reading this post? Check out our other articles.
Wilt u meer inzicht krijgen in uw Data? Contacteer ons nu
Wordt meer datagedreven met BiteStreams en laat de concurrentie achter je.Contacteer ons