Do we need Terraform for Helm application deployment?
I was recently asked to help migrate the Penterep.io project to a cloud provider. One of the few requirements for the deployment was the ability to start with a small platform and grow rapidly with coming customers. The other requirement was the ability to migrate applications among cloud providers easily. Since all ported apps were stateless, using Kubernetes for container orchestration was a no-brainer (scalability and portability are a strong suit of Kubernetes).
Deployment architecture has split into two stages (similarly as described here¹):
- infrastructure deployment — cloud resources orchestration (Kubernetes cluster, databases …) via Terraform
- Helm packages orchestration
This split is not arbitrary but is there for several reasons:
- It is not recommended² to create a resource provider in the same module where its resources are created (in other words, it is not advised to create a Kubernetes cluster with the same Terraform code as, e.g., its pods). It’s technically possible, but some actions don’t behave correctly (from my own experience, it behaves the same as described in the warning box).
- Infrastructure resources might be used by multiple Helm applications with an independent lifecycle.
- Even with a single Helm app, you probably don’t want to rotate your infrastructure resources when completely redeploying your application.
Helm packages orchestration
One of the project’s key goals was the portability of created solution, which meant most code in the Kubernetes/Helm (both of them are, in theory, independent of the used cloud provider) and as little code as possible in Terraform, which is on the other hand platform specific.
With legacy deployments, I’ve usually used Terraform for Let’s Encrypt certificates and DNS records management. With Kubernetes, we have better options. Certificate management is covered by cert-manager, and certificates can be easily created like this:
Creating DNS records is even easier with external-dns and these lines of code in annotations of ingress controller (or LB if you like):
The good old days of bribing DNS guys to have records ready in hours instead of days are luckily over.
After all this anabasis, I’ve noticed only Terraform resource left was helm_release. Is there any benefit in running Terraform for the deployment of a single Helm package instead of directly interacting with Helm? Let’s summarize the key features of Terraform:
- Orchestration of cloud resources — Terraform is excellent in automatically creating dependency tree among managed resources, leading to simple deployment code and fast parallel build even for large infrastructures.
- Execution plan — list of changes between the Terraform code and the state of managed infrastructure that will be applied with Terraform execution. In most cases, the actual state of infrastructure is fetched and compared to the current code.
While we don’t need Terraform to orchestrate a single resource, the way the execution plan is created for helm_release resources is downright annoying. For helm_release, Terraform detects only changes in Helm values or Chart version. In the case of deploying the chart directly from Terraform directory, no modification to Helm templates is detected and applied.
Since there is no clear advantage of using another deployment layer, terraform plan can be easily replaced with helm diff upgrade command, and terraform apply with helm upgrade --install. The additional benefit of this approach is a little bit faster deployment.
Troubles with the cloud provider
The approach above will work well for major cloud providers. In the case of OVH (our initial provider of choice), we faced issues both with the cert-manager and the external-dns.
Cert-manager (specifically Let’s Encrypt) uses so-called challenges to prove that you are the domain owner. The two types of challenges are mostly used. The more common HTTP challenge and the less known DNS challenge (you need the write access permissions to the DNS zone), which is the only option for certificates with wildcard domains. And as you would guess, upstream OVH code doesn’t support DNS challenges.
OVH integration in the external-dns is, on the other hand, not maintained at all. Trying to solve this issue³ took a few hours of my life and was show stopper for us.
Both of these issues forced us to manage certificates and DNS records with Terraform (OVH integration is excellent), so there was no reason not to use it for Helm application deployment also.
Final thoughts
In general, with Kubernetes, you should be able to create portable code that can be easily deployed to any cloud provider without using additional deployment tools. With a smaller cloud provider and a ton of bad luck, you will be thankful for fantastic tools like Terraform.
PS: Penterep.io is currently available and has multiple tiers, including a free one. If you wonder, what it can do for you, then it’s a web platform for comprehensive penetration testing. From the beginning, the platform was developed with a primary focus on modularity, scalability, and optimization. We created a considerably large environment, which is gradually being extended by other parts, and its possibilities of use are thus increasing over time. The main contribution of the platform is the unification of all penetration tester tasks into one place, automation of testing processes, and team collaboration between testers and other users. The result is a solution that improves the effect of penetration testing to such an extent that the time, complexity, and work required to complete the entire test successfully will be significantly lower than using the currently available tools.