Integrating Terraform and Helm using helm_release

An experience using Terraform and the helm_release resource

Terraform, Helm, and Kubernetes are all powerful tools for managing and deploying containerized applications on cloud environments. Terraform is an infrastructure-as-code (IaC) tool that allows you to create, manage, and version your cloud infrastructure. Helm, the package manager for Kubernetes, makes it easy to install, upgrade, and manage Kubernetes applications. And Kubernetes itself is an open-source container orchestration system that automates the deployment, scaling, and management of containerized applications. Combining these three tools allows you to automate the deployment and scaling of your applications on a Kubernetes cluster while maintaining the flexibility of your infrastructure.

In the last decade, IaC has made a profound impact on the industry. It is a practice that has proven itself, just like containers and Kubernetes, which are here to stay. Combining these practices in the form of Kubernetes, Terraform and Helm, can however be challenging. In this blog post, we will explore the different aspects of the Terraform helm_release module, and propose an alternative that might work better for your workflow.

An introduction of helm_release

The helm_release resource allows one to install a set of Kubernetes charts using the Helm tool, via a Terraform resource. What the resource does behind the scenes is using the helm upgrade command to either install or upgrade your chart.

An example usage of the helm_release resource for a local chart is the following:

1
2
3
4
5
6
7
8
9

resource "helm_release" "example" {
  name       = "my-local-chart"
  chart      = "./charts/example"
  
  set {
    name  = "worker.workerMemory"
    value = var.worker_memory
  }
}

Such a chart could be installed with Terraform, with a Helm provider installed and access configured, for example with a kubectl configuration:

1
2
3
4
5

provider "helm" {
  kubernetes {
    config_path = var.kube_config_path
  }
}

Terraform swallows all output

Conceptually this is great, as you can resort to just one tool to deploy and manage your infrastructure and applications. There are, however, several challenges using the helm_release resource.

When preparing for an upgrade, you naturally use the terraform plan command to check the changes you have made. This is the first challenge when using helm_release, as we cannot see any diff or output in terms of changes for the Helm charts. The only output we ever get to see is:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

  # module.api.helm_release.example will be updated in-place
  ~ resource "helm_release" "example" ***
        id                         = "example"
        name                       = "./charts/example"
        # (26 unchanged attributes hidden)

      - set ***
          - name  = "worker.workerMemory" -> null
          - value = "16Gi" -> null
        ***
      + set ***
          + name  = "worker.workerMemory"
          + value = "16Gi"
        ***
    ***

Plan: 0 to add, 1 to change, 0 to destroy.

As you can see in this example, the value of the Helm variable worker.workerMemory has not changed, but still we got a diff indicating that it changed to null and back to 16Gi. (This is an altered example of an output from a real CI/CD pipeline where the number of variables is much larger than this. We get 200 line diffs every single time for things that have not changed. Note that the diff does not show up if nothing changed at all.)

When something goes wrong with Helm

If the combination of the Helm template and the Terraform variables it received created an error, terraform apply will hang indefinitely. The command will give no output whatsoever on why the deployment failed.

A shortened example of this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

  # module.api.helm_release.example will be updated in-place
  ...

Plan: 0 to add, 1 to change, 0 to destroy.
module.api.helm_release.example: Modifying... [id=example]
module.api.helm_release.example: Still modifying... [id=example, 10s elapsed]
...
module.api.helm_release.example: Still modifying... [id=example, 5m20s elapsed]
╷
│ Error: timed out waiting for the condition
│ 
│   with module.api.helm_release.example,
│   on ../../modules/example/main.tf line 14, in resource "helm_release" "example":
│   14: resource "helm_release" "example" ***

Once this happens you can use the helm CLI tool to do some debugging, but this is definitely not the best experience.

When something needs to change

When something has to change in the Helm chart, it is not always the best experience deploying these changes. At times Terraform/Helm do not pick up the changes made in the Helm charts. The declarative approach of Terraform and Helm is a godsend, but if it does not work you will waste a lot of valuable time fixing the bug. One could try creating and destroying Helm charts at your behest, assuming that recreating them would solve these problems. However, two issues come up in this scenario:

1. The Helm charts are managed by Terraform, if we make any changes outside of Terraform we are in for trouble, as that means Terraform and its state will get out-of-sync.
2. Within Terraform we can directly alter (create/delete) specific Terraform resources. However, this also means that that is all we can do: we cannot change a particular aspect of a Helm chart. This is especially a problem if you have one parent Helm chart and subcharts, which is actually a great way to package your Helm charts.

An alternative approach to Helm & Terraform

Given the experiences described above, you might want to consider not managing your Helm charts with Terraform. However, all is not lost. A simple, yet effective approach to combining Helm & Terraform within one CI/CD pipeline does exist.

We could set this up as a two-step-approach. First, we make sure our Terraform changes have been made. Second, we avoid using helm_release and use Helm natively to update our applications and Kubernetes infrastructure. This way we get the best out of both worlds, and can still natively interface and debug with our Terraform and Helm software. Moreover, as our Kubernetes applications might depend on resources that we have built in the Terraform step, we can send the variables that Helm needs in the second step.

Let’s look at an example of the second step:

1
2
3
4

helm upgrade example ../../example -f values.yaml \
  --set api.postgres_instance_connection_name="$(terraform output -raw api_postgres_connection_name)" \
  --set worker.workerMemory="$(terraform output -raw worker_memory)" \
  --install

This approach effectively means that we instead call the two tools separately, possibly linking them together using shell scripts.

In this snippet we upgrade our example chart (and install if needed) but can still have one place where we manage our variables: Terraform. As an example, we pass in the postgres connection string into our Helm chart, which might come from the managed postgres instance (from your cloud providers) you built earlier using Terraform code.

To integrate this into a GitHub action workflow, we could build a CI/CD pipeline for deployments:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78

name: Deployment of Terraform & Helm to staging or prod.

on:
  push: # We do CD when pushing

jobs:
  deploy:
    name: Run terraform & helm
    defaults:
      run:
        working-directory: ./infra/environments/${{ github.ref == 'refs/heads/main' && 'prod' || 'staging' }}
    runs-on: ubuntu-latest
    concurrency: # Override previous deployment if we start a new one
      group: ${{ github.workflow }}-terraform

    steps:
      - name: Checkout code
        uses: actions/checkout@v2

      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v1

      - name: Terraform Format
        id: fmt
        run: terraform fmt -check

      - name: Terraform Init
        id: init
        run: terraform init

      - name: Terraform Validate
        id: validate
        run: terraform validate -no-color

      - name: Terraform Plan
        id: plan
        run: terraform plan -no-color -input=false
        continue-on-error: true

      - uses: actions/github-script@v6
        if: github.event_name == 'pull_request'
        env:
          PLAN: "terraform\n${{ steps.plan.outputs.stdout }}"
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          script: |
            const output = `#### Terraform Format and Style 🖌\`${{ steps.fmt.outcome }}\`
            #### Terraform Initialization ⚙️\`${{ steps.init.outcome }}\`
            #### Terraform Validation 🤖\`${{ steps.validate.outcome }}\`
            #### Terraform Plan 📖\`${{ steps.plan.outcome }}\`
            <details><summary>Show Plan</summary>
            \`\`\`\n
            ${process.env.PLAN}
            \`\`\`
            </details>
            *Pushed by: @${{ github.actor }}, Action: \`${{ github.event_name }}\`*`;
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: output
            })

      - name: Terraform Plan Status
        if: steps.plan.outcome == 'failure'
        run: exit 1

      - name: Terraform Apply
        if: (github.ref == 'refs/heads/staging' || github.ref == 'refs/heads/main') && github.event_name == 'push'
        run: terraform apply -auto-approve -input=false

      - name: Helm upgrade
        if: (github.ref == 'refs/heads/staging' || github.ref == 'refs/heads/main') && github.event_name == 'push'
        run: |
          helm upgrade example ../../example -f values.yaml \
              --set api.postgres_instance_connection_name="$(terraform output -raw api_postgres_connection_name)" \
              --set worker.workerMemory="$(terraform output -raw worker_memory)" \
              --install

This would perform terraform plan everytime you push to GitHub, and perform a deployment when merging or pushing to the staging or main branch.

To conclude: Helm and Terraform

Terraform, Helm, and Kubernetes are very powerful technologies, however, integrating these in nice workflows to be employed in CI/CD pipelines can get complicated. Here we have presented some of our learnings on this topic. We hope that this helps others on their journey of integrating technologies and automating their software pipeline. Happy coding!