Templating with Linux in a Shell Script?

Question

What I want to acomplish is:

1. Having a config file as template, with variables like $version $path (for example apache config)

2. Having a shell script that "fills in" the variables of the template and writes the generated file to disk.

Is this possible with a shell script. I would be very thankfull if you can name some commands/tools I can accomplish this or some good links.

Answer 1

This is very possible. A very simple way to implement this would be for the template file to actually be the script and use shell variables such as

#! /bin/bash
version="1.2.3"
path="/foo/bar/baz"
cat > /tmp/destfile <<-EOF
here is some config for version $version which should
also reference this path $path
EOF

You could even make this configurable on the command line by specifying version=$1 and path=$2, so you can run it like bash script /foo/bar/baz 1.2.3. The - before EOF causes whitespace before the lines be ignored, use plain <<EOF if you do not want that behavior.

Another way to do this would be to use the search and replace functionality of sed

#! /bin/bash
version="1.2.3"
path="/foo/bar/baz"
sed -e "s/VERSION/$version/g" -e "s/PATH/$path/" /path/to/templatefile > /tmp/destfile

which would replace each instance of the strings VERSION and PATH. If there are other reasons those strings would be in the template file you might make your search and replace be VERSION or %VERSION% or something less likely to be triggered accidentally.

Answer 2

The easiest way to do this simply in Linux CLI is to use envsubst and Environment Variables.

Example template file apache.tmpl:

<VirtualHost *:${PORT}>
    ServerName ${SERVER_NAME}
    ServerAlias ${SERVER_ALIAS}
    DocumentRoot "${DOCUMENT_ROOT}"
</VirtualHost>

Run envsubst and output result to new file my_apache_site.conf:

export PORT="443"
export SERVER_NAME="example.com"
export SERVER_ALIAS="www.example.com"
export DOCUMENT_ROOT="/var/www/html/"
envsubst < apache.tmpl > my_apache_site.conf

Output:

<VirtualHost *:443>
    ServerName example.com
    ServerAlias www.example.com
    DocumentRoot "/var/www/html/"
</VirtualHost>

Answer 3

No tools necessary other than /bin/sh. Given a template file of the form

Version: ${version}
Path: ${path}

or even with mixed shell code included

Version: ${version}
Path: ${path}
Cost: ${cost}\$
$(i=1; for w in one two three four; do echo Param${i}: ${w}; i=$(expr $i + 1); done)

and a shell parsable configuration file like

version="1.2.3-r42"
path="/some/place/under/the/rainbow/where/files/dance/in/happiness"
cost="42"

it is a simple matter to expand this to

Version: 1.2.3-r42
Path: /some/place/under/the/rainbow/where/files/dance/in/happiness
Cost: 42$
Param1: one
Param2: two
Param3: three
Param4: four

Indeed, given the path to the configuration file in shell variable config_file and the path to the template file in template_file, all you need to do is:

. ${config_file}
template="$(cat ${template_file})"
eval "echo \"${template}\""

This is perhaps prettier than having complete shell script as the template file (@mtinberg's solution).

The complete naive template expander program:

#!/bin/sh

PROG=$(basename $0)

usage()
{
    echo "${PROG} <template-file> [ <config-file> ]"
}

expand()
{
    local template="$(cat $1)"
    eval "echo \"${template}\""
}

case $# in
    1) expand "$1";;
    2) . "$2"; expand "$1";;
    *) usage; exit 0;;
esac

This will output the expansion to standard output; just redirect standard output to a file or modify the above in obvious fashion to produce the desired output file.

Caveats: Template file expansion would not work if the file contained unescaped double quotes ("). For security reasons, we should probably include some obvious sanity checks or, even better, perform shell escaping transformation if the template file is generated by external entity.

Answer 4

You probably ought to look into a configuration management system like Puppet or Chef. These can easily do what you describe above and much more.

Answer 5

If you want lightweight and real templates rather than shell code that generates new files, the usual choices are sed& awk or perl. Here is one link: http://savvyadmin.com/generate-text-from-templates-scripts-and-csv-data/

Me, I'd use a real language like perl, tcl, python, ruby or something else in that class. Something built for scripting. They all have good, simple templating tools and tons of examples in google.

Answer 6

I use shtpl for that. (private project of mine, which means, it is not widely in use. But maybe you want to test it anyway)

For example you want to generate an /etc/network/interfaces out of a csv-file, you can do it like that:

CSV-file content (here test.csv):

eth0;10.1.0.10;255.255.0.0;10.1.0.1
eth1;192.168.0.10; 255.255.255.0;192.168.0.1

Template (here interfaces.tpl):

#% IFS=';'
#% while read "Val1" "Val2" "Val3" "Val4"; do
auto $Val1 
iface $Val1 inet static
  address $Val2 
  netmask $Val3 
  gateway $Val4 

#% done < "$CSVFILE"

Command:

$ CSVFILE=test.csv sh -c "$( shtpl interfaces.tpl )"

Result:

auto eth0 
iface eth0 inet static
  address 10.1.0.10 
  netmask 255.255.0.0 
  gateway 10.1.0.1 

auto eth1 
iface eth1 inet static
  address 192.168.0.10 
  netmask  255.255.255.0 
  gateway 192.168.0.1

Enjoy!

Answer 7

I improved FooF's answer so that the user doesn't need to unescape double quotes manually:

#!/bin/bash
template="$(cat $1)"
template=$(sed 's/\([^\\]\)"/\1\\"/g; s/^"/\\"/g' <<< "$template")
eval "echo \"${template}\""

Answer 8

Blatant Promotion Ahead

So while researching existing shell-based template engines, I came across this post.

I'd like to throw my creation into the ring for consideration:

Bash-TPL | https://github.com/TekWizely/bash-tpl

A smart, lightweight shell script templating engine, written in Bash

Bash-TPL lets you you mark up textual files (config files, yaml, xml, scripts, html, etc) with shell commands and variable replacements, while minimally impacting your original file layout.

Templates are compiled into shell scripts that you can invoke (along with variables, arguments, etc) to generate complete and well-formatted output text files.

So in the context of the OP's question, a template might look like this:

test.tpl

Version: <% $version %>
Path: <% $path %>

A simple example of executing the template with some data:

$ version="v1.0.0" path="/path/to/the/thing" source <( bash-tpl test.tpl )

Version: v1.0.0
Path: /path/to/the/thing

This simple example is just to show how Bash-TPL can answer the OP's question.

The template engine itself supports many more features.

I hope you'll give it a try if you're in the market for a smart, easy to use shell script template engine.

Answer 9

I am probably late to this party. However, I stumbled about the very same problem and I opted for creating my own BASH template engine in a few lines of code:

Lets say that you have this file.template:

# My template
## Author
 - @NAME@ <@EMAIL@>

And this rules file:

NAME=LEOPOLDO WINSTON
EMAIL=leothewinston\@leoserver.com

You execute this command:

templater rules < file.template

You get this:

# My template
## Author
 - LEOPOLDO WINSTON <[email protected]>

You can install it by:

 bpkg install vicentebolea/bash-templater

This is the project site

Answer 10

I recently published a bash script that accomplishs just that using a jinja-like template syntax. It's called cookie. Here's a demo:

Answer 11

To expand on @FooF's great answer (newlines didn't format in comment), using a heredoc + control characters, you can permit arbitrary characters and filenames:

template() {
    # if [ "$#" -eq 0 ] ; then return; fi # or just error
    eval "cat <<$(printf '\x04\x04\x04');
$(cat $1)
"
}

This accepts any non-null character, and only truncates early if 3 ^D bytes are encountered on their own line (realistically never). zsh even supports null terminators, so printf '\x00\x00\x00' would work. template even works for treacherous filenames like:

for num in `seq 10`; do
    template 'foo "$ .html' # works
done

Beware shell templates can "expand" arbitrary commands, e.g. $(launch_nukes.sh --target $(curl -sL https://freegeoip.app/csv/ | cut -d, -f 9,10)). With great power…

Edit: if you really don't want your files launching nukes at you, just sudo -u nobody sh (or some other safe user) beforehand.

Answer 12

Perhaps I may be able to pique your interest on a script above all other scripts that have, thus far, been suggested here.

Given a template.txt:

Hello, {{person}}!

Kindly execute:

$ person=Bob ./render template.txt

And you shall see the output

Hello, Bob!

You may write it to a file by redirecting stdout as follows:

$ person=Bob ./render template.txt > rendered.txt

Or declare your variables in a file and then source it. This is best done inside a script:

#!/usr/bin/env bash
source ./myvalues
./render template.txt > rendered.txt

An advantage of said renderer above all the others is that it does not expand any variables such as $foo or ${bar} which is quite useful if what you are endeavoring to render is, itself, a script! Furthermore, said renderer is unit tested and has received pull requests from the community.

I have been told that I must declare that I have written this script myself. Thus, I shall mention that I wrote it and is available in GitHub at https://github.com/relaxdiego/renderest

Some script stats if that interests you:

-------------------------
| Effective LOCs |  20  |
| Test Coverage  | 100% |
| Test LOCs      |  50  |
-------------------------

Answer 13

I created a shell templating script also named shtpl (just like zstegi, I didn't know, sorry!). My shtpl uses a jinja-like syntax which, now that I use ansible a lot, I'm pretty familiar with:

$ cat /tmp/test
{{ aux=4 }}
{{ myarray=( a b c d ) }}
{{ A_RANDOM=$RANDOM }}
$A_RANDOM
{% if $(( $A_RANDOM%2 )) == 0 %}
$A_RANDOM is even
{% else %}
$A_RANDOM is odd
{% endif %}
{% if $(( $A_RANDOM%2 )) == 0 %}
{% for n in 1 2 3 $aux %}
\$myarray[$((n-1))]: ${myarray[$((n-1))]}
/etc/passwd field #$n: $(grep $USER /etc/passwd | cut -d: -f$n)
{% endfor %}
{% else %}
{% for n in {1..4} %}
\$myarray[$((n-1))]: ${myarray[$((n-1))]}
/etc/group field #$n: $(grep ^$USER /etc/group | cut -d: -f$n)
{% endfor %}
{% endif %}


$ ./shtpl < /tmp/test
6535
6535 is odd
$myarray[0]: a
/etc/group field #1: myusername
$myarray[1]: b
/etc/group field #2: x
$myarray[2]: c
/etc/group field #3: 1001
$myarray[3]: d
/etc/group field #4: 

More info on my github

Answer 14

Quick and dirty solution that safely supports full shell scripting.

template.sh

#!/bin/sh

EOF=EOF
exec cat <<EOF | sh
cat <<EOF
$(cat $1 | \
    sed 's|`|\\`|g' | \
    sed 's|\$|\\\$|g' | \
    sed "s|${OPEN:-<%}|\`eval echo |g" | \
    sed "s|${CLOSE:-%>}|\`|g")
$EOF
EOF

Simply run the following on any file you want to template.

sh template.sh file.txt.tmpl

file.txt.tmpl

uname: <% $(uname) %>
pwd: <% `pwd` %>
ls: $(ls)
df: `df`
SHELL: <% $SHELL %>
HOME: <% ${HOME} %>
USER: $USER
PATH: ${PATH}

The output would be something like the following.

uname: Linux
pwd: /home/clayrisser
ls: $(ls)
df: `df`
SHELL: /usr/bin/zsh
HOME: /home/clayrisser
USER: $USER
PATH: ${PATH}

If you want to write the templated file, simply redirect the output to a new file.

sh template.sh file.txt.tmpl > file.txt

You can change the open and close characters by setting OPEN and CLOSE.

file.txt.tmpl

uname: {{ $(uname) }}
pwd: {{ `pwd` }}
ls: $(ls)
df: `df`
SHELL: {{ $SHELL }}
HOME: {{ ${HOME} }}
USER: $USER
PATH: ${PATH}

OPEN="{{" CLOSE="}}" sh template.sh file.txt.tmpl