Infrastructure as Code (IaC): Resource Provisioning with AWS Cloudformation

Infrastructre as Code (IaC) is an extremely important topic. It is where you create an entire infrastructure of hardware and software from a single point. The process of creating the required resources and services is called Resource Provisioning. This is the first step for creating a system. The next step is Configuration Management where you configure the resources and services you created the way you want them (install software and things). Further steps exist down the IaC lane. However this article will describe Resource Provisioning only. This article is more of a concept illustration than a detailed introduction to Cloudformation.

Note that it is expected you know some cloud technology, I will be using AWS Cloudformation on the Amazon Web Services cloud. You may use any other cloud provider, they all provide their respective infrastructre as code services. Cloudformation is a declarative way of specifiying which resources/services you want across the cloud. Virtually any resource on AWS can be created using CLoudformation. It is free to use but you pay for the resources you use.

AWS also provides the Cloud Development Kit (CDK) which is an object oriented way of provisioning resources. Both CDK and CLoudformation do the same thing at the end. Cloudformation uses Json/Yaml to declare your resources whereas CDK uses a SDK in the supported languages. CDK provides more flexibility by using programming language constructs. Using either one is not the matter of this article. But one last thing to note is the fact that Cloudformation gets pretty long pretty quickly.

I will be using Yaml as opposed to Json due to clarity.

Why provision resources centrally:

Why go through the hassle? Imagine you got an infrastructure where you keep working on. You then want to expand it and potentially scale it. The complexity starts getting unmanagable by one person. A person makes mistakes. A person forgets the allocated resources. Now suddenly you got an executive order to expand the offering to another region, you will now need to reprovision everything, it takes days to do that. More mistakes/bugs are introduced.

Now notice how much time was allocated above? Maybe some corners were cut along the way to just get the system to production. This is a real issue. That’s where IaC shines. There is only a single source of truth to your entire infrastructure. And you can include it in Git and use it with your CI/CD pipeline.

Another reason maybe to make different environments for your system. e.g. Development, Staging and Production.

AWS Cloudformation has virtually no limits and behind the scenes it is transaction like. That is either all resources are created or nothing which is really important.

When you update the Cloudformation stack you create, only changed resources are updated. If an update fails, it will rollback to the previous successful state.

It is testable. You can provision the infrastructure directly and test each thing and find it exists. This may cost you. Other less costly solutions exist such as LocalStack which are even better in some cases.

When you delete a stack, all provisioned resources will be deleted. Deletion policy exists which allows you to retain some resources you choose.

Provisioning an EC2 instance:

To provision an EC2 instance, is simple enough. With Cloudformation you provision the bare minimums unlike the console which does some heavy lifting. I expect you know some things about EC2. Will create a cloudformation.yaml file (the name is not an issue):

Resources:
 Server:
 Type: AWS::EC2::Instance
 Properties:
 BlockDeviceMappings:
 - DeviceName: "/dev/sda1"
 Ebs: 
 VolumeSize: 10
 VolumeType: gp3
 DeleteOnTermination: true
 ImageId: ami-07fbdcfe29326c4fb
 InstanceType: t2.micro

Here I define an EC2 instance named Server. Note that the name is not the name of the EC2 instance but the name here is so you reference this resource throughout the template. Note that the EC2 instance is defined under Resources. This is the only required component of a stack template. Others exist which will be discussed later. Note that every resource must have a predefined type which can be found here: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-template-resource-type-ref.html

Each resource has properties and the respective properties are defined under the respective predefined resource. Click here for the EC2 type.

I have defined the EC2 instance with a gp3 EBS volume on a Linux AMI inside a t2.micro machine with 10GiB.

Now to run the instance. We can create the stack up in AWS console, use AWS CLI or any AWS SDK. I will use the CLI, I assume you have access to use Cloudformation and create/list EC2 instances in your IAM user.

aws cloudformation create-stack --stack-name basic --template-body file://cloudformation.yaml

Now this makes a new stack. Will return the stack id (ARN). Now stack creation is an asynchronous operation, thus will take it’s time to create. Give it some time and check what happened as below:

aws cloudformation describe-stacks --stack-name basic

Will return a JSON describing the stack including the current status and any tags attached to it.

Now I am pinging the EC2 instance using ping <ip-address>, I cannot get to it. It is because of EC2 security groups, I have not attached one to the instance. I got the EC2 instance IP from AWS Console, you can use the CLI.

Now I will update the cloudformation template to add a security group resource and attach it to the EC2 instance:

ServerSecurityGroup:
 Type: AWS::EC2::SecurityGroup
 Properties:
 GroupDescription: Server Security Group
 SecurityGroupIngress:
 - IpProtocol: icmp
 CidrIp: 0.0.0.0/0
 FromPort: -1
 ToPort: -1

And add the following to the EC2 instance:

SecurityGroups:
 - !Ref ServerSecurityGroup

And update the stack using the following command:

aws cloudformation update-stack --stack-name basic --template-body file://cloudformation.yaml

Now I can ping the instance easily. You need to remember that the IP changed since a new instance was launched since due to changing that resource’s properties. The -1 used above for ToPort and FromPort indicate any port.

The !Ref keyword is a special Cloudformation function which means reference to. It references another name in the template.

You may have some issues doing this. An issue may occur due to creating the instance before the security group because the instance depends on the security group. To resolve that we can add the following outside the Properties as:

DependsOn:
 - ServerSecurityGroup

If you just add this and update the stack, you will get an error since you really did not change any resource will do it’s magic when you create/update resource.

Description of a stack:

You know the Resources component of this template file. Now I want to add some description to the stack. To do so is easy, add Description at the same indentation level as the Resources:

Description: Hello world

When you query the stack using describe-stacks command, the description will be displayed.

Input parameters:

The same way you create infrastructure to your needs, customizability and flexibility is very important. To avoid hard coding any values, parameters are user input that you pass upon creation of the stack that may be used all throughout the stack. To define what parameters do the following (parameters are a top-level component so they are at the same level of Resources and Description):

Parameters:
 ServerType:
 Description: The EC2 instance type
 Type: String

Replace the InstanceType property to reference to the ServerType parameter:

InstanceType: !Ref ServerType

Now will update the stack the same way except now I include the parameters I am updating:

aws cloudformation update-stack --stack-name basic --template-body file://cloudformation.yaml --parameters ParameterKey=ServerType,ParameterValue="t2.micro"

Note when you create the stack, all parameters that have non-default values must be provided with the --parameters flag. Here I did not define a parameter with default value.

The reference to how parameters work is here: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/parameters-section-structure.html

Outputs:

Now another great thing is to output somethings from the stack creation. I want to output the ARN of the instance I am creating. This is so easily done in it’s own Outputs component as a top-level component:

Outputs:
 InstanceId:
 Value: !Ref Server

Outputs can also be used when working with multiple stacks/templates. The outputs are not shown unless the state of the stack is COMPLETE.

More here: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/outputs-section-structure.html

Other components:

Other components do exist in the Cloudformation template. ALl of them are described here by AWS: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-anatomy.html

The Serverless Application Model uses a simplified version of Cloudformation to define serverless components. The reason for simplification is since there is a lot of boilerplate code involved in setting up lambda functions and other associations.

Conclusion:

Now this was quite a ride. A lot of ground has been covered. For most basic software, a single stack is enough. However, more complicated systems do exist and these are possible by more advanced features with Cloudformation.

Infrastructure as Code is very useful indeed as no 2 developers can disagree on what was created when they look at an infrastructure code.

The Cloudformation template I made can be found here: https://github.com/Morr0/IaCArticles/blob/main/resource_provisioning/cloudformation.yaml

Thanks for reading through this article.