Infrastructure as a Code using Terraform Modules (Part-I)

Infrastructure is the base of any project irrespective of the field of the industry you are working in. So as in the software industry infrastructure is the most important thing. Having maintained infrastructure is necessary to run and test software builds and applications because it is the base of any application stack. And now when the software industry is pacing up and rolling new updates every week it becomes really important to configure and provision infrastructure. With configuration, we mean that the dependencies and libraries we will use to run our application should be regularly updated to figure out whether our application is still supporting the updated libraries or not. So maintaining and configuring infrastructure is important and it should be configured fluently to deploy regular updates of our application.

What is the hurdle?

In the decade of cloud-computing where we can use the services provided by different cloud providers such as AWS, Azure, GCP, etc. and will pay only for that which we used, it became quite cheap to set up an infrastructure. So now if someone wants to launch an application in the production he could use the cloud provider’s infrastructure and could launch his app.

But, will it be so easy for a developer to set up infrastructure on cloud in the first attempt. he would prefer to read the documentation and start setting up infrastructure from scratch. So let’s suppose he launched 3 instances for his app, a load balancer, Cloudwatch for monitoring, VPCs, Subnets, a DNS server, assigned security groups, and moving forward he ended up launching his infrastructure on the cloud. Now this infra which he designed in the stage environment, will need the same replica for the production. So it will be again tedious for him to do the same job and so will waste a lot of time and effort to set up the same for production. A cloud practioner workload will definitely increase in this situation and to meet the software delivery deadlines he may end up in technical failure. Every week cloud is releasing a bunch of new services and updates so it is not easy to have information about every resource and there are almost thousands of security vulnerabilities reported while setting up infrastructure manually. So building infrastructure is one thing and maintaining it forever is another big challenge.

What is Infrastructure as a Code?

To overcome the above situation software companies shifted to a different approach. Now practicing IAAS (Infrastructure as a service) they used IAAC (Infrastructure as a Code) approach to set up Infrastructure. Developers started to write code which when compiled will perform some sort of API calls in the cloud console and will set up the infrastructure in one go. Now instead of remembering documentation of operating all the services, you just need a piece of code to set up an Infrastructure. So now setting up the infrastructure is far way more simple and easy to implement. Also if any new service pops in it could be implemented by a piece of code. Also if a developer needs to deploy his application to the prod server he could just execute the same code which he developed for the staging and can set up his infrastructure. So until now, we can sense that implementing IAAC can bring robustness in building and provisioning the infrastructure. Infrastructure as a Service (IAAS) and Infrastructure as a Code(IAAC) implemented together can make infrastructure provisioning and configuration easier. Think of IAAS as a simple VM and OS provided by a Cloud service provider and IAAC as a file that will set up resources like subnets, VPC, Route 53, Security groups on the cloud.

Tools

There are many tools in the market which supports Infrastructure as a code. Over the past many years many tools have been designed to meet special intents but choosing the right tool for the right action is a required thing. Developers must know which tool will support the current scenario. There are many tools such as Chef, Puppet, Ansible, Terraform, etc. which can be used to provision infrastructure and configure environments. So I will discuss two different domains on which I will be able to fit these tools.

Configuration Orchestration:-

• Configuration Orchestration is the tool that is used to build and provision server instances themselves, making them run on the cloud.
• Then configure management tools could work and install certain dependencies and libraries used to run your application.
• Orchestration is not required when we work on small nodes or containers, there we can use simple configuration management tools.
• When we work on different clusters then we need an orchestration tool that keeps a check that all resources are working fine or not.
• If there is any issue the orchestrator tool will remove the damaged resource and will replace it with another.
• Terraform is considered as one of the best configuration orchestration tools which

Configuration Management:-

• Configuring Management is the tool that is used to install and manage the dependencies and software over the server instances.
• After the configuration orchestration tools have set up the instances and resources these tools then come to function by installing certain dependencies, packages, scripts, config files on each instance.
• These tools install dependencies on top of the infrastructure making it suitable to run our application.
• There are many configuration management tools such as Chef, Puppet, Ansible which work on different nodes and manage the configurations.

Terraform — An orchestration Tool

— — — — — — — — — — — — — — — — — — — — — — — — — — —Terraform is an open-source Hashicorp’s tool that supports cloud orchestration. With Terraform you can provision the resources on the cloud and launch as many as instances or resources as you need. All that Terraform needs is a .tf configuration file. This file is written in Terraform DSL (Domain-specific Language) and when executed it sends API calls to the cloud console and makes the resource available on the cloud. A simple example of Terraform file is shown below.

So in the first block, you will specify the cloud provider and region in which you want to work. After this block, you could define any resource in the resource block by mentioning the type of resource and resource name like here I have defined an aws_instance with the name of example. Then in the resource block, I have defined some basic variables as the ami which is the amazon image which you want to use and instance_type. And so in aws_instance, I have used one more tag block in which I have initialized the name instance. A point noted here is that it is a simple configuration file to just explain the syntax of the language which terraform uses. You can also see the documentation in which all types of resources are listed with their usage. Select your cloud provider and scroll down to the resources.

Okay, so let us run this file. Pre-requisites are that you need to have terraform installed in your local machine. If not installed check the steps here. Ok so now we are good to go. Open Terminal and navigate to the folder where you have your terraform configuration file( .tf ).

• Run terraform init. This command first checks whether there is a configuration file or not. If the file is present it initializes the backend means the environment on which terraform would run. Then it will download the plugins according to the provider you choose.

• Run terraform plan. This command tells us prior that which actions are gonna be executed and which resources are gonna made after we applied this code.
• Run terraform apply. This command will tell us the actions and resources which are gonna set up and asks for confirmation from the user. The + sign in the prefix of each resource states that these resources are gonna added. Answer YES and this command will start setting up the infrastructure. At last, if all goes good it will print the number of resources created.

• Similarly, if you need to destroy your instances write terraform destroy. This command will tell us the actions and resources which are gonna remove and asks for confirmation from the user. The — sign in the prefix of each resource states that these resources are gonna removed. Answer YES and this command will start destroying up the infrastructure. At last, if all goes good it will print the number of resources destroyed.

So as you can it successfully created the resource, you could also check your resource running on AWS or any cloud provider you are using.

Terraform Modules

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

So until now, we wrote a simple infrastructure code that will establish a single resource on the cloud. With Terraform files we could establish all the VPC, Subnets, Network gateways, Instances, and all other resources through simple code. But, As you manage your infrastructure with Terraform, you will create increasingly complex configurations. There is no intrinsic limit to the complexity of a single Terraform configuration file or directory, so it is possible to continue writing and updating your configuration files in a single directory. However, if you do, you may encounter one or more problems:

• Understanding and navigating the configuration files will become increasingly difficult.
• Updating the configuration will become riskier, as an update to one section may cause unintended consequences to other parts of your configuration.
• There will be an increasing amount of duplication of similar blocks of configuration, for instance when configuring separate dev/staging/production environments, which will cause an increasing burden when updating those parts of your configuration.
• You may wish to share parts of your configuration between projects and teams, and will quickly find that cutting and pasting blocks of configuration between projects is error-prone and hard to maintain.

As you know maintaining code is also very important and you should follow some better practices to make your code managed and reusable. So to maintain the infrastructure code and make it reusable in every environment we came up with the concept of Terraform Modules.

What is Terraform Module?

A Terraform module can be said as a directory which contains a bunch of configuration files which are arranged in such a way that one particular module will perform a specific task or function. Every file in a module is executed. A simple structure of the module is shown below:-

$ tree minimal-module/ . 
├── LICENSE 
├── README.md 
├── main.tf 
├── variables.tf 
├── outputs.tf

Calling Modules

Terraform commands will only work where there are configuration files. However, if you want to use this module in some other directory, then you need to use module block and specify the module in the source argument of the block. When you do terraform init it initializes the module and picks up the configuration files stored in that module.

Local and Remote Modules

You can either create your own modules or use the remote ones which are uploaded on the public terraform registry by many cloud source providers such as AWS, GCP, Oracle, Azure, or maybe some user also. The use of remote modules is that they are regularly maintained by cloud services and are completely working. You could either import your own created modules stored in a private registry.

Writing Modules is good Practice !!!!

As modules could be related to as a blueprint of the infrastructure creation, you should always write modules in order to make independent resources and make them reusable for any environment. Some of the good practices you should need to follow while writing modules are given below:-

• Use the public Terraform Registry to find useful modules. By reading and studying the verified modules you could actually gain a lot of confidence and could help a lot in writing your own module.
• Use local modules to organize and encapsulate your code. Even if you aren’t using or publishing remote modules, organizing your configuration in terms of modules from the beginning will significantly reduce the burden of maintaining and updating your configuration as your infrastructure grows in complexity.
• Publish and share modules with your team. Most infrastructure is managed by a team of people, and modules are an important way that teams can work together to create and maintain infrastructure. You could publish your modules in a public or a private registry according to the requirements.

Terraform Public Modules

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

Public terraform modules are always available on terraform registry. So now we will implement a way to use these private registry modules to provide an environment on the cloud. Well, the cloud which I have used here is AWS so you could do it on that or it is okay to use any cloud provider of your choice. Just remember to import the source link of the module according to the service provider.

• Open the page for terraform module registry for ec2 instance. At the top right-hand side, you will see the source code to import the module into your configuration file.

• Open a file named asmain.tf and create a module block named ec2-instance and write the source and version from the web page. Then you also have to initialize some variables in the module so that you could provision your infrastructure in your environment.

• In order to use most modules, you will need to pass input variables to the module configuration. The configuration that calls a module is responsible for setting its input values, which are passed as arguments in the module block. Aside from source and version.Some input variables are required, meaning that the module doesn’t provide a default value — an explicit value must be provided in order for Terraform to run correctly. Now create a variable.tf and initialize the variables which we used in the module.

• Modules also have output values, which are defined within the module with the output keyword. You can access them by referring to module.<MODULE NAME>.<OUTPUT NAME>. Like input variables, module outputs are listed under the outputs tab in the terraform registry.Create an output.tf file and make the output variables initialized of your own choice or just copy the below file.

Provision Infrastructure

• Run terraform init and initialize your terraform configuration. So it will first download the modules from the source which you initialized in your configuration file (Mine is with the name main.tf ).

• Run terraform plan to know beforehand that everything is working fine or are there any errors. If there is an error fix it.

• Run terraform apply to apply the changes in the cloud. Before applying changes terraform will ask about the confirmation. Say yes to confirm your changes.

• You will see that resources will be created and it will print the output variables which we initialized in the output.tf.

• Run terraform destroy to destroy changes in the cloud. Before destroying changes it will ask confirmation to destroy the infrastructure on the cloud. Answer yes to confirm it.

• You will see that your resources will be destroyed from the cloud console as well.

So by using the pre-written modules, we could provision our infrastructure.

Working Of Modules

Lets now study how this scenario actually worked. Go to your workspace and open .terraform folder. It would be hidden so you will not be able to list it normally in Linux (use ls -a to list all files). In the .terraform folder, you will see that there is a subfolder named as modules. Open the folder and you will see the following structure.

├── ec2_module
│   └── terraform-aws-ec2-instance-2.13.0
│       ├── CHANGELOG.md
│       ├── examples
│       │   ├── basic
│       │   │   ├── main.tf
│       │   │   ├── outputs.tf
│       │   │   └── README.md
│       │   └── volume-attachment
│       │       ├── main.tf
│       │       ├── outputs.tf
│       │       └── README.md
│       ├── LICENSE
│       ├── main.tf
│       ├── Makefile
│       ├── outputs.tf
│       ├── README.md
│       └── variables.tf
└── modules.json

Open modules/terraform-aws-ec2-instance-2.13.0/main.tf file and you will see the actual resources defined there. So when you imported the module it compiled all these files in the backend and provisioned your infrastructure. So by using modules, you don’t write actual configuration files but you supply the variables defined in the module and run it. See the pre-written configuration file main.tf below:-

$ cat ec2_instance/terraform-aws-ec2-instance-2.13.0/main.tf 
locals {
  is_t_instance_type = replace(var.instance_type, "/^t(2|3|3a){1}\\..*$/", "1") == "1" ? true : false
}
resource "aws_instance" "this" {
  count = var.instance_count
ami              = var.ami
  instance_type    = var.instance_type
  user_data        = var.user_data
  user_data_base64 = var.user_data_base64
  subnet_id = length(var.network_interface) > 0 ? null : element(
    distinct(compact(concat([var.subnet_id], var.subnet_ids))),
    count.index,
  )
  key_name               = var.key_name
  monitoring             = var.monitoring
  get_password_data      = var.get_password_data
  vpc_security_group_ids = var.vpc_security_group_ids
  iam_instance_profile   = var.iam_instance_profile
associate_public_ip_address = var.associate_public_ip_address
  private_ip                  = length(var.private_ips) > 0 ? element(var.private_ips, count.index) : var.private_ip
  ipv6_address_count          = var.ipv6_address_count
  ipv6_addresses              = var.ipv6_addresses
ebs_optimized = var.ebs_optimized
dynamic "root_block_device" {
    for_each = var.root_block_device
    content {
      delete_on_termination = lookup(root_block_device.value, "delete_on_termination", null)
      encrypted             = lookup(root_block_device.value, "encrypted", null)
      iops                  = lookup(root_block_device.value, "iops", null)
      kms_key_id            = lookup(root_block_device.value, "kms_key_id", null)
      volume_size           = lookup(root_block_device.value, "volume_size", null)
      volume_type           = lookup(root_block_device.value, "volume_type", null)
    }
  }
dynamic "ebs_block_device" {
    for_each = var.ebs_block_device
    content {
      delete_on_termination = lookup(ebs_block_device.value, "delete_on_termination", null)
      device_name           = ebs_block_device.value.device_name
      encrypted             = lookup(ebs_block_device.value, "encrypted", null)
      iops                  = lookup(ebs_block_device.value, "iops", null)
      kms_key_id            = lookup(ebs_block_device.value, "kms_key_id", null)
      snapshot_id           = lookup(ebs_block_device.value, "snapshot_id", null)
      volume_size           = lookup(ebs_block_device.value, "volume_size", null)
      volume_type           = lookup(ebs_block_device.value, "volume_type", null)
    }
  }
dynamic "ephemeral_block_device" {
    for_each = var.ephemeral_block_device
    content {
      device_name  = ephemeral_block_device.value.device_name
      no_device    = lookup(ephemeral_block_device.value, "no_device", null)
      virtual_name = lookup(ephemeral_block_device.value, "virtual_name", null)
    }
  }
dynamic "network_interface" {
    for_each = var.network_interface
    content {
      device_index          = network_interface.value.device_index
      network_interface_id  = lookup(network_interface.value, "network_interface_id", null)
      delete_on_termination = lookup(network_interface.value, "delete_on_termination", false)
    }
  }
source_dest_check                    = length(var.network_interface) > 0 ? null : var.source_dest_check
  disable_api_termination              = var.disable_api_termination
  instance_initiated_shutdown_behavior = var.instance_initiated_shutdown_behavior
  placement_group                      = var.placement_group
  tenancy                              = var.tenancy
tags = merge(
    {
      "Name" = var.instance_count > 1 || var.use_num_suffix ? format("%s-%d", var.name, count.index + 1) : var.name
    },
    var.tags,
  )
volume_tags = merge(
    {
      "Name" = var.instance_count > 1 || var.use_num_suffix ? format("%s-%d", var.name, count.index + 1) : var.name
    },
    var.volume_tags,
  )
credit_specification {
    cpu_credits = local.is_t_instance_type ? var.cpu_credits : null
  }
}

Likewise, there is variable.tf and output.tf file in which we have declared the variables and outputs respectively. When we call the module in our configuration we actually initialize the variables written in this variable.tf file. Also, if we need the output we could only get the output of those variables which are defined in this output.tf file.

Now if you have the infrastructure source code in the .terraform folder you could change it and improvise it to your needs. You don’t this resource okay remove the block and execute it. So with modules you can actually change or configure your infrastructure.

Similarly, you can reuse these modules in different environments and set up your infrastructure in seconds. Create a module block, add the source and version, initialize some required variables, execute it, and you are done. So within seconds, you could actually create a production environment by using the source code which you used to provision the stage environment.

So it is considered as the best practice to write your own Terraform modules and maintain them so as to build Infrastructure with the help of code. Maintaining, Reusing, Configuring, Customizing infrastructure has become way so simple by writing Terraform Modules. So yeah, with this I conclude my Part-I and in Part-II, we will see actually how to create a terraform module and publish it in private and public registries.

Any queries you could ping me to my mail id or you could also follow me on Twitter.