Terraform Synced-State Validation

Working with Terraform creates a number of exciting challenges.

In this blog we are going to focus on two of them:

1. Keeping Terraform’s state consistent with the actual provider (i.e. a cloud provider) where resources can be quite a challenge. Such inconsistencies are usually a result of changes performed directly in the cloud environment.
2. Keeping the Terraform code synced with the Terraform state.  When we want to apply changes in Terraform, we can:

• Start by pushing the changes to the Terraform repository and then apply them. The Terraform state will be behind until the changes are affected.
• Start by applying the changes locally, then push them. The Terraform state will be ahead until the code is pushed.

The first option can be mostly avoided if we don’t allow manual actions that could have been performed by Terraform, but not entirely.

The real issue with the second option is that after the first action (apply/push), Terraform will be out of sync. If the second action is delayed or not happening, the undesired inconsistent state will be kept.

To ensure eventual consistency here, you can automate the workflow with an automation framework, such as Jenkins.

However, the Fyber DevOps team, wanted to avoid such an automation process because we thought it would become a delaying factor, and we still manage to keep Terraform state synced with the code and the actual resources.

We do this with a little help from a Jenkins pipeline library script. The script initials, updates and executes a ‘Terraform Plan’ command. If Terraform isn’t synced, it reports the unsynced components to the dedicated Slack channel. The Jenkins job is scheduled to run in the first and penultimate working hour, on each of the Terraform environments.  This enables us to be able to fix any inconsistency by the end of the day.

It’s important to note that the script is useful when you’re working with the Terraform Recommended Workflow, specifically when having an environment file for each environment that defines the Terraform modules, while the resources lie in their own repository.