26th July 2018

When teams start to work with Terraform, especially in POC-type projects, sometimes secrets are directly hardcoded in Terraform files in order to have results quickly. However, we all know that this is not best practice at all. Having hardcoded secrets in Terraform files means that secrets are available to any user with access to the code repository, thus exposing system access to a wider pool of users than needed. Not to mention, this then also gives access to potential malicious users, who could get access to source code repository where Terraform files are stored and versioned.

The aim of this post is to show you how to store secrets in Azure Key Vault and use them in your Terraform configurations by using a simple integration via Terraform external data source. The objective of the following set of scripts is to show you the concept and the way to retrieve any secret from Azure Key Vault to use it in your Terraform configurations.

For the sake of simplicity, here our Terraform configuration is kept to a minimal setup, where a resource group is built and the client secret is used in order to authenticate against Azure, and the build that the resource comes from is Azure Key Vault. Any Terraform flow will require to authenticate Terraform against Azure, so it can manage resources in our subscriptions. Then the entry point is the client secret. We will use it to provide access to Azure Key Vault, and from that point, the secret will be used but from the Key Vault. The following diagram shows visually how secrets are transferred between the different actors and reflect conceptually the processes involved.

The following is assumed. Feel free to adapt these values to a more suitable use case for you:

  • Azure subscription
  • Azure Key Vault named kvault-kv  
  • Terraform
  • Powershell with AzureRm cmdlets available
  • Service Principal so Terraform can interact with Azure. Service Principal must have permissions to read the Key Vault.

Prepare scripts to retrieve secrets

Copy and create a new script with the following code. Update it with the right tenant ID.

function ReadInput() {
 # Read JSON payload from stdin
 $jsonpayload = [Console]::In.ReadLine()
 $params = ConvertFrom-Json $jsonpayload
 $csecret = $params.secret
 $clientId = $params.clientId

function Login() {
 $SecurePassword = $csecret | ConvertTo-SecureString -AsPlainText -Force
 $cred = new-object -typename System.Management.Automation.PSCredential -argumentlist $clientId, $SecurePassword
 $tenantid = “xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx”
 Connect-AzureRmAccount -ServicePrincipal -Credential $cred -TenantId $tenantid | Out-Null

function GetSecrets() {
 $secrets = Get-AzureKeyVaultSecret -VaultName $kvName
 $json = “{”
 foreach ($secret in $secrets) {
   $temp = Get-AzureKeyVaultSecret -VaultName $kvName  -Name $secret.Name
   $attr = $temp.Name
   $value = $temp.SecretValueText
   $json += “`”” + $attr + “`” : `”” + $value + “`”,”
 $json = $json.Substring(0, $json.Length – 1)
 $json += “}”

$kvName = “kvault-kv”
Write-Output $json

This script will allow you to connect to your Azure account and retrieve secrets from the Key Vault. The output is a string that contains a JSON object with all your secrets.


Setup Terraform configuration

Let’s apply this to the very first secret needed in Terraform, the client secret. Store a secret in Azure Key Vault using the name azure-clientsecret and your own client secret as value.

Consider the following Terraform stack that will just build a new resource group. Amend them accordingly to you subscription and naming preferences.

provider “azurerm” {
 subscription_id = “${var.subscription_id}”
 client_id       = “${var.client_id}”
 client_secret   = “${var.client_secret}”
 tenant_id       = “${var.tenant_id}”

#Create a resource group
resource “azurerm_resource_group” “myrg” {
 name     = “${var.resource_group_name}”
 location = “${var.location_name}”

with the corresponding variables.tf file

variable “subscription_id” {}
variable “client_id” {}
variable “client_secret” {}
variable “tenant_id” {}
variable “resource_group_name” {}
variable “location_name” {}

and vars.tfvars file

subscription_id = “your-subscription-id”
client_id = “your-client-id”
client_secret = “your-not-so-secret-client-secret”
tenant_id = “your-tenant-id”
resource_group_name = “compute-dev-rg”
location_name = “NorthEurope”


Integrate with Azure keyvault

In this example we will use the client secret twice, and each time the source of that secret will be different. The client secret is the entry point in order to start working with our Terraform stack. This secret will be provided in command line in this example.

The first thing to do is to remove your client_secret line from vars.tfvars file.

Modify your main.tf file so the result is the following:

data “external” “secrets” {
 program = [“powershell.exe”, “./retrieveSecrets.ps1”]
 query = {
   secret   = “${var.client_secret}” # Secret comes from command line
   clientId = “${var.client_id}”

provider “azurerm” {
 subscription_id = “${var.subscription_id}”
 client_id       = “${var.client_id}”
 client_secret   = “${data.external.secrets.result.azure-clientsecret}” # Secret comes from Azure Keyvault
 tenant_id       = “${var.tenant_id}”

#Create a resource group
resource “azurerm_resource_group” “myrg” {
 name     = “${var.resource_group_name}”
 location = “${var.location_name}”

An external data source resource has been added. In this case, this example runs on a Windows machine and PowerShell is being used to retrieve the secrets from Key Vault.

Everything is ready and now it is time to initialise our working folder:  terraform init

Get Terraform plan:  terraform plan -var-file=”vars.tfvars” -var “client_secret=xxxxxxxxx”

This way, we are providing the client secret so the retrieveSecrets.ps1 script can connect to Azure and read from the Key Vault. After the execution of the external data source, the client secret used to create the resource group comes from Azure Key Vault.

If the plan is correct, apply it:  terraform apply -var-file=”vars.tfvars” -var “client_secret=xxxxxxxxx”

You should have now a new resource group in your subscription and Terraform used the client secret stored in Key Vault to build it.

Now the idea is to apply the same pattern to all other secrets like VM passwords, etc:

  • Create the secret in Azure Key Vault with a name of your choice
  • Replace the value in your Terraform file(s) with ${data.external.secrets.result.<name-used-in-keyvault>}

That’s it! With this approach you will be consuming all your secrets from Azure Key Vault.

In this post we have started with a Terraform configuration that had the client secret as a variable exposed. Then we have removed the secret from Terraform files and stored it Azure Key Vault. We are using the simplest approach to provide our script access to Azure and we are providing the secret also via command line. Once that initial step is done, the rest of the process uses the secret that comes from Azure Key Vault. Extending this approach to other secrets in our configurations is immediate and running our Terraform deployments from a CI/CD tool that can manage credentials to store our client secret for the firs step will make a more complete and realistic use case.

This is a simple solution that will allow you to deliver your project using better practices from the very beginning.

Happy and safe Terraforming in Azure!!

About Liam McDowell
Liam tries to keep our clients happy. He is a complete control freak so don't touch his suite of pens, cutlery or dare to sit in his chair!