Bashing your YAML

Have you ever needed to fill in a YAML template without adding any other tools other than your friendly neighborhood BASH?

Here is how you can do it.

Let’s start with a YAML file with BASH variables in it.

----
# ${sample}
nats:
  password: "${nats_password}"
  username: '${nats_username}'
  machines: ${nats_machines}

Cut and paste the above YAML snippet into a file called template.yml.

The only characters you need to be concerned with inside the data are the backslash ‘\’, dollar ‘$’, and backquote, or if you prefer, backtick ‘`’. The dollar and backquote characters are used to introduce the parameter or command substitution syntax. The backslash is because that character escapes the previous two characters.

Here is the secret sauce to process this YAML file with BASH.

export sample="Hello Yaml!"
export nats_machines="10.2.3.4"
export nats_username="nats"
export nats_password="password"
rm -f final.yml temp.yml
( echo "cat <<EOF >final.yml";
  cat template.yml;
  echo "EOF";
) >temp.yml
. temp.yml
cat final.yml

Cut and paste the above code into a file named process.sh. Change the permissions so process.sh is executable.

Now run process.sh.

./process.sh

Your output should look like:

---
# Hello Yaml!
nats:
  password: "password"
  username: 'nats'
  machines: 10.2.3.4

The nice part is that files with quotes are handled correctly.

We can also handle files like PEM files, though here we also use AWK, which is usually on boxes with BASH

---
# ${sample}
nats:
  password: "${nats_password}"
  username: '${nats_username}'
  machines: ${nats_machines}
bosh_exporter:
  bosh:
    ca_cert: |
$(awk '{printf "      %s\n", $0}' < cert.pem)

Now let’s add the PEM file and name it cert.pem.

--- BEGIN ---
Four score and seven
beers ago, our forefathers
....
--- END ---

We don’t need to change the process.yml file, so run process.sh again should get the following output.

---
# Hello Yaml!
nats:
  password: "password"
  username: 'nats'
  machines: 10.2.3.4
bosh_exporter:
  bosh:
    ca_cert: |
      --- BEGIN ---
      Four score and seven
      beers ago, our forefathers
      ....
      --- END ---

This was tested in BASH 3.2 and 4.4.

Notes:

1. There is an existing bug in BASH that the source command does not support process substitution. There is a workaround but the code seems less readable than using the temporary file approach above. Here is the code in case you disagree.

   source /dev/stdin  <<<"$(echo 'cat <final.yml'; cat template.yml; echo EOF;)"
   

2. There is a way to do PEM files without resorting to AWK, but again it seems less readable and has one issue. If your last line in the PEM file does not have a newline, the read loop below will drop the last line.

   while read line
   do
       printf "    %s\n" "${line}"
   done <cert.pem