After covering the basics for Terraform modules and how to use them locally in the last post, let’s now dive into the implementation of the remote version in detail and some thoughts about module philosophy.
Remote modules
As the name suggests, remote Modules are located in a separate Terraform configuration in a different location and not in the same directory. There are several ways to reference the created modules from another location.
For example, the modules can be hosted in another Git repository or in Azure DevOps. However, an AWS S3 bucket, Google Storage or any other storage location can also be selected, as long as access is possible. Remote modules are useful for creating and maintaining shared infrastructure modules that can be reused in different projects without reinventing the wheel each time.
In this post, we’ll focus on Git repositories. However, Mercurial, for example, is also supported via Bitbucket. Git is certainly the most prominent option for module versioning.
Protocol Choice
The question remains to be clarified as to which protocol should be used to access it. SSH or HTTPS are available. In practice, HTTPS is often used because it is easier to handle than configuring SSH connections. Instructions for referencing via SSH, including private-public key pair generation and the correct storage locations of the keys, can be found at[1].
For the use of remote modules, a separate branch in the existing repository was created. The module code itself is moved to a new repository, which is available at[2]. The local folder with the resources for the web app has been transferred. Now only the source value has to be adjusted.
The syntax of referencing is slightly different, it needs a module URL:
A small stumbling block lurks in front of the folder name of the remote module, here you absolutely need two “//“. Once the URL has been entered, a terraform init command is still missing, as the module must first be loaded just like a new provider:
For referencing via SSH, SSH is used instead of HTTPS in the source URL[3].
Versioning of modules
The big advantage of remote modules is the fact that URL parameters can be used to reference certain branches and versions by tags. This allows the modules to be further developed and tested independently of each other without influencing existing configurations. For local modules in the directory, only the current version could be used.
To illustrate the example – and to follow the best practice anyway – therefore a branch in the new module Repository was created where the adjustments are made.
The change was made in the feature/demochange branch. The new source configuration will then be:
Of course, the change is only available after it has also been published to the remote repository.
Furthermore, there is the possibility to address a specific tag in the repository. Tags are assigned in the development process, for example, when a release has been created. As an simple example, a first version in Github was published:
The referencing would then look like this:
Instead of the tag, a concrete commit ID can also be addressed:
The main reason to work this way is not a new insight: new features or changes to existing implementations should take place independently of production environments. Only when the developments are validated and tested, especially if they are breaking changes, should they find their way into the master branch for production. This is the only way to work effectively in distributed teams.
Whenever the value of the source attribute or variable compositions change in the module call, the init command must be re-executed so that Terraform can reload the module from the source. Otherwise, we are greeted by the following error message:
Some caution is always required with the changes in remote modules. Even though a change has been committed and pushed to the remote branch, Terraform may not be aware of it. If another terraform init does not help, the .terraform folder can be deleted manually. After that, Terraform securely loads the new version via init. The problem only exists with deployments via the local machine, such legacy issues are usually not found in a usual DevOps pipeline setup if you have new worker nodes for the runs.
Remote Special: Azure Module of the Public Terraform Registry
It’s worth taking a look at another source of existing remote repositories with modules: the Azure Modules on terraform.io[4]. These are modules published by Microsoft. These bundle resources for Azure Kubernetes Service, virtual networks or Postgres databases, for example. As long as the implementations are suitable for your own needs, this can save time and effort. The modules are versioned and have a detailed description of the resources used and input configurations. A subset of these are the Azure Terraform verified modules. These have gone through a separate review and testing process in the Microsoft pipelines[5]. Pull requests for the modules are also accepted from the development community. To do this, a few standards must be met in terms of naming, structure and tests[6].
Module Philosophy
After finalizing the technical details, here are also a few thoughts about the creation of modules which cover observed the aspects described below or similar situations in a number of projects.
Many infrastructure teams want to deploy a standardized IaC repository for Terraform in the enterprise. There is a lot of philosophizing about how the modules should be structured. The result is often a library of very granular modules with an overloaded parameter list that try to catch every special case. This is probably due to the well-known way of thinking from object-oriented programming languages. However, since Terraform and IaC are generally not a traditional language, this approach often fails. Some of the module libraries do not get a stable, mature version and the effort was in vain, because the whole thing will be tried again in the next project. In order to create useful and manageable modules, it should be questioned what exactly is to be achieved.
Define objectives precisely
Usually, when encapsulating methods, one tries to make them as independent as possible. Each method is supposed to fulfill exactly one function.
The same can be said about modules. Terraform modules are more likely to be understood as predefined, related configurations. So you don’t create a single resource, but aim to provide a usable infrastructure. It should therefore be considered in advance what use the module should have for the end user.
A classic example is a web server module. Let’s assume that a VM is deployed within the module. A web server is installed on the VM and the module returns the server URL and deployment URL as output parameters. Such a module is very useful. Which module is likely to be used less is one that only creates a VM. If this module is used, then probably only in conjunction with a web server installer.
In principle, the following points should be considered when defining modules:
- Encapsulation – Infrastructure that is always deployed together should be grouped together in one module
- Consider ownership – It is important to define who is responsible for maintaining each component. A team should be responsible for all the resources in a module.
- Differentiation according to lifecycle – Long-lived and short-lived infrastructure should be outsourced to separate modules.
When is it worth creating a module?
The question often arises as to whether it is worthwhile to create a module. In the Terraform documentation[7] you can find the following flowchart:
If you follow the different paths, the pattern of developing modules for a specific use case is striking. Modules that only create a single resource are usually not worthwhile. Such modules usually degenerate into simple instantaneous water heaters for parameters. In practice, nothing was gained and the code was bloated in the process.
The next part of the series will cover thoughts about cloud agnostic modules and best practices in module usage.
[1] https://www.theserverside.com/blog/Coffee-Talk-Java-News-Stories-and-Opinions/github-clone-with-ssh-keys
[2] https://github.com/thomash0815/wd-tfseries-v2/tree/main
[3] https://developer.hashicorp.com/terraform/language/modules/sources
[4] https://registry.terraform.io/namespaces/Azure
[5] https://github.com/Azure/terraform-azure-modules
[6] https://github.com/Jingwei-MS/terraform-azure-modules/blob/main/Contribute.md
[7]https://developer.hashicorp.com/terraform/cloud-docs/recommended-practices/part3.2#3-create-your-first-module