The Good, the Bad, and the Ugly of Using Terraform-Compliance

Lately, I’ve been focusing on DevOps, pipelines, and Terraform. In this article I explore Terraform-Compliance, and reveal what’s good, not so good, and downright confusing about this tool.

Firstly, for reference, Terraform-Compliance is…

a lightweight, security and compliance focused test framework against Terraform to enable negative testing capability for your infrastructure-as-code

Unlike the other tools that I’ve tested and written articles about (namely: Checkov, TFSec, and the GitHub Super-Linter), Terraform-Compliance approaches scanning in a different way. The former tools listed just require you to point to the directory where your Terraform files exist. But with Terraform-Compliance, it requires a Terraform Plan file or the Terraform State file to run against.

Obviously, to produce a .tfplan / .plan file, or have a .tfstate file, that means you need to actually execute your Terraform code. That also means, whatever your target environment is (ie. Azure, AWS, GCP, etc.), will need to be accessible as part of your pipeline.

Behavior-Driven Development (BDD)

To level-set, Terraform-Compliance uses a framework that utilizes a behavior-driven development approach and leverages Radish to process this. This is something new to me, but the Wikipedia article explains it this way:

BDD is largely facilitated through the use of a simple domain-specific language (DSL) using natural-language constructs (e.g., English-like sentences) that can express the behaviour and the expected outcomes.

Without getting into all the details (which are explained on the BDD Reference page), this uses a pattern of…

Given I have <resource_type> defined
When it contains <some_property>
Then its value must be <pattern>
And its value must match the “<pattern>” regex

Here is a simple example for an Azure Storage Account test…

Given I have azurerm_storage_account defined
Then it must have enable_https_traffic_only
And its value must be true

Obviously, that is a lot easier to read and understand (as a human), when working with testing/validation.

The Good

What I like about Terraform-Compliance, is the clear and understandable output that you receive when the tests complete. For example, from the screenshot below, it’s clear what it’s checking for (ie. the ‘enable_https_traffic_only‘ property), and the value it expects (ie. ‘true‘).

As we can also see in the output, when using Terraform-Compliance in a CI/CD pipeline, it will produce a non-zero exit code if there are any failed tests.

Terraform-Compliance – Completed Scan (Failed Status)

The Bad

Although I’ve worked with several other Terraform scanning tools (as mentioned at the beginning of this article), Terraform-Compliance is different in that it requires additional effort to work with it. It’s not a simple “point it to my Terraform files directory” and let it go.

You Have To Plan

Firstly, as mentioned at the outset, Terraform-Compliance requires a Terraform Plan file or the Terraform State file to execute against. But in reality, that’s not too big of a deal, it just means you have to be able to successfully execute Terraform INIT and Terraform PLAN from your pipeline.

Terraform Plan -Out File

It’s not really a “bad” thing, but it is something you need to take into account, especially if you’re using other tools that just scan all the .TF files without the need for compiling.

Limited Examples

If you read the Terraform-Compliance documentation, in particular for the Behavior-Driven Development (BDD) Features examples, there’s not a lot of them. On the Examples page, there is only 23 listed. But, if you look at the accompanying terraform-compliance/user-friendly-features GitHub repo, there are 20 examples for AWS, and 5 examples for Azure. However, some of the Feature examples contain multiple tests. For example, the Azure Database.feature contains 3 scenarios that it checks for, and the Azure AppService.feature has 6 scenarios.

Still, there currently is a limited set of example Features to work from. But, that can be easily resolved by you, the community! If you start to use Terraform-Compliance, and building your own Features, you can (and should) contribute those back to the greater community.

What is a ‘Feature’?

One other thing I found unclear and confusing, was the concept of a Feature, how it works, and how to create them myself. In the BDD Reference documentation, it does walk you through the various elements the make up a BDD Feature (ie. GIVEN, WHEN, THEN, etc.).

TerraformCompliance – BDD Feature – Example

But what it doesn’t do, is show how to actually create one! It wasn’t until I looked around the GitHub repository, and at the specific examples, that I realized that the file-type was “.feature”! Something as simple as that should be explained up-front when explaining Behavior-Driven Development (BDD). We should also be provided with a simple framework/template structure to work from. Other languages (ie. Terraform, Azure Resource Manager (ARM) Templates, etc.), all have beginner/starter templates to help people learn how to use the language. The Behavior-Driven Development (BDD) Features documentation should be the same.

The Ugly

Now we get into some of the less-than-positive experiences. Just to be clear, these are not “show stoppers” that should prevent/deter you from using Terraform-Compliance. These are just some really confusing things I ran into, that made my experience frustrating.

Terraform Versions

The first ugly thing I encountered was when using Terraform-Compliance in a Docker container. To be clear, this is a supported method for this tool. And the Usage documentation explains all of the command-line interface (CLI) parameters you can use.

However, what is not documented, is what version of Terraform the tool is using (or how often/frequently this is updated). This is an issue because, you can only find out what version of Terraform you must use by running (and failing) the scan. Case in put (see screenshot), in my pipeline I installed the latest version of Terraform, then ran Terraform INIT and PLAN to produce the required .plan file. When sending this .plan file as an input to Terraform-Compliance (via the –planfile plan_file parameter), I was presented with the message shown below.

TerraformCompliance – Invalid Plan File Terraform Version

That’s not a great end-user experience. Especially since the end-user may have a reason to use an older version of Terraform (ie. they are not ready to upgrade yet, or more testing is needed before they commit to a newer version of Terraform), or a reason to use a newer version of Terraform (ie. they are using some new feature that is only available in the latest version).

In fact, Terraform-Compliance does not support end-user control over what version of Terraform is used! Case in point, at the time of this writing, this is an open issue (Issue#365 Multiple versions of terraform executable in the Docker image).

Workaround

As a workaround (if you’re using the Docker container method), to discover which version of Terraform is present, you will need to just run the container once.

Even though the Docker file has an argument for LATEST_TERRAFORM_VERSION, there is no indication where/how that gets set. It would be useful if there was a way we could check which Terraform version is targetted. That would be better than having to run the container and check for failure.

Of note, if you install Terraform-Compliance natively (ie. not use Docker container method), you will not encounter this issue. This is because, although not apparent from the documentation, Terraform-Compliance uses the Terraform EXE just to convert the TFPLAN file into a plan.out.json file, nothing else! So, you could also perform that conversion yourself after your Terraform PLAN command, like this:

terraform show -json plan.out > plan.out.json

…and then use that .JSON file as the input for Terraform-Compliance. Since it won’t need to convert anything, it will execute without issue.

Extracting Output

The other “ugly” thing I found, was trying to extract the results from Terraform-Compliance. If you’ve read any of my other blog posts (Checkov, TFSec, GitHub Super-Linter), you will know what my goal is. Basically the whole focus on running these types of tools is not only to scan my Terraform code for quality and security issue, but also to incorporate that into an Azure DevOps CI/CD pipeline, and present those results as a Pipeline Test Result.

Using the methods I’ve used with the other referenced tools, I first attempted the following (using StdOut):

docker run --rm --volume $(System.DefaultWorkingDirectory)/Path-That-Has-Your-TFPlan-File:/target --interactive eerkunt/terraform-compliance --features git:https://github.com/terraform-compliance/user-friendly-features.git --planfile plan.out > $(System.DefaultWorkingDirectory)/TFCompliance-Report.xml

But, unfortunately that didn’t work. In the pipeline I saw the following error:

##[warning]Failed to read /home/vsts/work/1/s/TerraformCompliance/TFCompliance-Report.xml. Error : Data at the root level is invalid. Line 1, position 1..

And the resulting XML file looks something like this:

TerraformCompliance – StdOut – XML File

If you’ve read my Publishing Checkov Terraform Quality Checks to Azure DevOps Pipelines article, those square characters (ie. ” “) may look familiar. But the challenge is, we don’t know the exact format the Behavior-Driven Development (BDD) output is produced in. In fact, there is no documentation on –output options at all, even though (if you look into the code), the silent_formatter.py file makes reference to Cucumber (which it itself has a reference to JUnit Output Formatter).

Discovery

So, after reaching out to the author (Emre Erkunt), he indicated that Terraform-Compliance supports all parameters that ‘radish-bdd’ uses, one of which is ‘–junit-xml’. He also pointed me to an Issue on the GitHub repo that addressed this very need (Issue#271 Export JUnit XML Report format).

So, in short, we can modify the Terraform-Compliance execution code to include “–junit-xml” as follows:

docker run --volume $(System.DefaultWorkingDirectory)/Path-That-Has-Your-TFPlan-File/:/target --interactive eerkunt/terraform-compliance --junit-xml TFCompliance-Report.xml --features git:https://github.com/terraform-compliance/user-friendly-features.git --planfile plan.out

 

With that supported (but undocumented) output option, I can then capture the XML file and publish the results in my Azure DevOps Pipeline as Test Results.

TerraformCompliance – Azure DevOps Pipeline – Test Results

Conclusion

With this clearer understanding of how to use Terraform-Compliance, and Behavior-Driven Development (BDD) Features, it is a useful tool in your Infrastructure-as-Code (IaC) toolbox.

Obviously, there are several key elements (ie. how Feature files work (and how to author them), output options, Terraform version references/controls, etc.) that are currently lacking in the documentation. However, I have been in touch with the author, and he has agreed to add these missing elements to the documentation, to improve the learning/use experience of this tool.

And don’t forget, if you start to author your own Feature tests, think about contributing them back to the community for others to use as examples.