53

What I want to acomplish is:

  1. Having a config file as template, with variables like $version $path (for example apache config)

  2. Having a shell script that "fills in" the variables of the template and writes the generated file to disk.

Is this possible with a shell script. I would be very thankfull if you can name some commands/tools I can accomplish this or some good links.

14 Answers 14

41

This is very possible. A very simple way to implement this would be for the template file to actually be the script and use shell variables such as

#! /bin/bash
version="1.2.3"
path="/foo/bar/baz"
cat > /tmp/destfile <<-EOF
here is some config for version $version which should
also reference this path $path
EOF

You could even make this configurable on the command line by specifying version=$1 and path=$2, so you can run it like bash script /foo/bar/baz 1.2.3. The - before EOF causes whitespace before the lines be ignored, use plain <<EOF if you do not want that behavior.

Another way to do this would be to use the search and replace functionality of sed

#! /bin/bash
version="1.2.3"
path="/foo/bar/baz"
sed -e "s/VERSION/$version/g" -e "s/PATH/$path/" /path/to/templatefile > /tmp/destfile

which would replace each instance of the strings VERSION and PATH. If there are other reasons those strings would be in the template file you might make your search and replace be VERSION or %VERSION% or something less likely to be triggered accidentally.

2
  • 1
    This will become much more challenging if your variables contain newlines
    – chiborg
    Jul 11, 2021 at 13:22
  • To escape dollar sign in the template, use backslash: \$ Mar 25, 2022 at 18:22
59

The easiest way to do this simply in Linux CLI is to use envsubst and Environment Variables.

Example template file apache.tmpl:

<VirtualHost *:${PORT}>
    ServerName ${SERVER_NAME}
    ServerAlias ${SERVER_ALIAS}
    DocumentRoot "${DOCUMENT_ROOT}"
</VirtualHost>

Run envsubst and output result to new file my_apache_site.conf:

export PORT="443"
export SERVER_NAME="example.com"
export SERVER_ALIAS="www.example.com"
export DOCUMENT_ROOT="/var/www/html/"
envsubst < apache.tmpl > my_apache_site.conf

Output:

<VirtualHost *:443>
    ServerName example.com
    ServerAlias www.example.com
    DocumentRoot "/var/www/html/"
</VirtualHost>
1
  • 7
    It's part of GNU gettext, and therefore available on all major Unix platforms (including macOS X), as well as Microsoft Windows.
    – tricasse
    Mar 30, 2019 at 23:28
27

No tools necessary other than /bin/sh. Given a template file of the form

Version: ${version}
Path: ${path}

or even with mixed shell code included

Version: ${version}
Path: ${path}
Cost: ${cost}\$
$(i=1; for w in one two three four; do echo Param${i}: ${w}; i=$(expr $i + 1); done)

and a shell parsable configuration file like

version="1.2.3-r42"
path="/some/place/under/the/rainbow/where/files/dance/in/happiness"
cost="42"

it is a simple matter to expand this to

Version: 1.2.3-r42
Path: /some/place/under/the/rainbow/where/files/dance/in/happiness
Cost: 42$
Param1: one
Param2: two
Param3: three
Param4: four

Indeed, given the path to the configuration file in shell variable config_file and the path to the template file in template_file, all you need to do is:

. ${config_file}
template="$(cat ${template_file})"
eval "echo \"${template}\""

This is perhaps prettier than having complete shell script as the template file (@mtinberg's solution).

The complete naive template expander program:

#!/bin/sh

PROG=$(basename $0)

usage()
{
    echo "${PROG} <template-file> [ <config-file> ]"
}

expand()
{
    local template="$(cat $1)"
    eval "echo \"${template}\""
}

case $# in
    1) expand "$1";;
    2) . "$2"; expand "$1";;
    *) usage; exit 0;;
esac

This will output the expansion to standard output; just redirect standard output to a file or modify the above in obvious fashion to produce the desired output file.

Caveats: Template file expansion would not work if the file contained unescaped double quotes ("). For security reasons, we should probably include some obvious sanity checks or, even better, perform shell escaping transformation if the template file is generated by external entity.

2
4

You probably ought to look into a configuration management system like Puppet or Chef. These can easily do what you describe above and much more.

4
  • 1
    thanks. Absolutely, I have Chef installed and running. But it adds a lot of overhead, when you have to write your own cookbooks. I do not know the ruby programming language and my conclusion was. its easier to do this with a shell script for the easier cases (if possible).
    – Markus
    Jul 6, 2011 at 21:42
  • Looks like Puppet and Chef both use ERB for templates, and that's ridiculously easy to get started with. Given a variable name, the string <%= name %> in a template will get replaced with name's value. How you define name outside the template differs between the two systems, obviously. Jul 6, 2011 at 23:42
  • 1
    Yes templating (With Chef) itself is absolutely easy. But using chef as Framework (and writing the cookbooks) requires a lot of time. To get the data into the templates you need to understand where and how Chef manages the "merging" of datasources and a lot of other stuff. I have started writing my own cookbooks, but a shell script would in my special case be 100 times faster...
    – Markus
    Jul 15, 2011 at 18:49
  • 1
    Getting the infrastructure going for Chef or Puppet can be a pain or you can try to figure out how to run them headless which is a fun adventure. Ansible works fine in pull or push mode out of the box so it might strike a better balance of figuring it out and scripting it yourself. docs.ansible.com/template_module.html
    – chicks
    Jun 16, 2015 at 13:27
4

If you want lightweight and real templates rather than shell code that generates new files, the usual choices are sed& awk or perl. Here is one link: http://savvyadmin.com/generate-text-from-templates-scripts-and-csv-data/

Me, I'd use a real language like perl, tcl, python, ruby or something else in that class. Something built for scripting. They all have good, simple templating tools and tons of examples in google.

4

I use shtpl for that. (private project of mine, which means, it is not widely in use. But maybe you want to test it anyway)

For example you want to generate an /etc/network/interfaces out of a csv-file, you can do it like that:

CSV-file content (here test.csv):

eth0;10.1.0.10;255.255.0.0;10.1.0.1
eth1;192.168.0.10; 255.255.255.0;192.168.0.1

Template (here interfaces.tpl):

#% IFS=';'
#% while read "Val1" "Val2" "Val3" "Val4"; do
auto $Val1 
iface $Val1 inet static
  address $Val2 
  netmask $Val3 
  gateway $Val4 

#% done < "$CSVFILE"

Command:

$ CSVFILE=test.csv sh -c "$( shtpl interfaces.tpl )"

Result:

auto eth0 
iface eth0 inet static
  address 10.1.0.10 
  netmask 255.255.0.0 
  gateway 10.1.0.1 

auto eth1 
iface eth1 inet static
  address 192.168.0.10 
  netmask  255.255.255.0 
  gateway 192.168.0.1

Enjoy!

0
2

I improved FooF's answer so that the user doesn't need to unescape double quotes manually:

#!/bin/bash
template="$(cat $1)"
template=$(sed 's/\([^\\]\)"/\1\\"/g; s/^"/\\"/g' <<< "$template")
eval "echo \"${template}\""
2

Blatant Promotion Ahead

So while researching existing shell-based template engines, I came across this post.

I'd like to throw my creation into the ring for consideration:

Bash-TPL | https://github.com/TekWizely/bash-tpl

A smart, lightweight shell script templating engine, written in Bash

Bash-TPL lets you you mark up textual files (config files, yaml, xml, scripts, html, etc) with shell commands and variable replacements, while minimally impacting your original file layout.

Templates are compiled into shell scripts that you can invoke (along with variables, arguments, etc) to generate complete and well-formatted output text files.

So in the context of the OP's question, a template might look like this:

test.tpl

Version: <% $version %>
Path: <% $path %>

A simple example of executing the template with some data:

$ version="v1.0.0" path="/path/to/the/thing" source <( bash-tpl test.tpl )

Version: v1.0.0
Path: /path/to/the/thing

This simple example is just to show how Bash-TPL can answer the OP's question.

The template engine itself supports many more features.

I hope you'll give it a try if you're in the market for a smart, easy to use shell script template engine.

1
  • This is a GREAT little bit of shell code! It only took a few minutes to use it to create a complex Dockerfile template that made it SO more readable and easy to maintain. The .INCLUDE directive is particularly handy and can be used to create complex "nested" templates/code. If you are strictly working in Bash then really stop looking, just use this, forget about all the other templating possiblites.
    – DKebler
    Apr 13 at 1:25
1

I am probably late to this party. However, I stumbled about the very same problem and I opted for creating my own BASH template engine in a few lines of code:

Lets say that you have this file.template:

# My template
## Author
 - @NAME@ <@EMAIL@>

And this rules file:

NAME=LEOPOLDO WINSTON
EMAIL=leothewinston\@leoserver.com

You execute this command:

templater rules < file.template

You get this:

# My template
## Author
 - LEOPOLDO WINSTON <[email protected]>

You can install it by:

 bpkg install vicentebolea/bash-templater

This is the project site

1

I recently published a bash script that accomplishs just that using a jinja-like template syntax. It's called cookie. Here's a demo:

cookie demo

4
  • How does this answer the question?
    – RalfFriedl
    Nov 14, 2018 at 6:45
  • 1
    @RalfFriedl I'm not sure what you mean. I just reread the question thinking I must have missed something, but I'm not seeing it. In fact, how does this not answer any part of the question that was asked? It's almost like I built cookie with the sole purpose of answering this question... which I kindof did, though I may have worded it a bit differently in my head at the time. :) Nov 14, 2018 at 10:54
  • I'm a little late to play mediator, but perhaps Ralf was just charitably prompting you to explain how cookie answers the question, rather than trying to uncharitably imply you don't know your own project well enough to judge whether it might solve someone's problem.
    – abathur
    Apr 25, 2019 at 0:28
  • 1
    Is it just me, or those animated gifs are almost impossible to follow?.. At least you need to watch them several times. No way to stop, move backwards. Experience akin to the marquee tag.
    – x-yuri
    Feb 5, 2021 at 22:20
1

To expand on @FooF's great answer (newlines didn't format in comment), using a heredoc + control characters, you can permit arbitrary characters and filenames:

template() {
    # if [ "$#" -eq 0 ] ; then return; fi # or just error
    eval "cat <<$(printf '\x04\x04\x04');
$(cat $1)
"
}

This accepts any non-null character, and only truncates early if 3 ^D bytes are encountered on their own line (realistically never). zsh even supports null terminators, so printf '\x00\x00\x00' would work. template even works for treacherous filenames like:

for num in `seq 10`; do
    template 'foo "$ .html' # works
done

Beware shell templates can "expand" arbitrary commands, e.g. $(launch_nukes.sh --target $(curl -sL https://freegeoip.app/csv/ | cut -d, -f 9,10)). With great power…

Edit: if you really don't want your files launching nukes at you, just sudo -u nobody sh (or some other safe user) beforehand.

0

Perhaps I may be able to pique your interest on a script above all other scripts that have, thus far, been suggested here.

Given a template.txt:

Hello, {{person}}!

Kindly execute:

$ person=Bob ./render template.txt

And you shall see the output

Hello, Bob!

You may write it to a file by redirecting stdout as follows:

$ person=Bob ./render template.txt > rendered.txt

Or declare your variables in a file and then source it. This is best done inside a script:

#!/usr/bin/env bash
source ./myvalues
./render template.txt > rendered.txt

An advantage of said renderer above all the others is that it does not expand any variables such as $foo or ${bar} which is quite useful if what you are endeavoring to render is, itself, a script! Furthermore, said renderer is unit tested and has received pull requests from the community.

I have been told that I must declare that I have written this script myself. Thus, I shall mention that I wrote it and is available in GitHub at https://github.com/relaxdiego/renderest

Some script stats if that interests you:

-------------------------
| Effective LOCs |  20  |
| Test Coverage  | 100% |
| Test LOCs      |  50  |
-------------------------
2
  • You should disclose that you wrote this script. And maybe write your answer a little more ... professionally. Jul 22, 2020 at 19:22
  • @GeraldSchneider done! Thank you for reviewing. Jul 23, 2020 at 22:52
0

I created a shell templating script also named shtpl (just like zstegi, I didn't know, sorry!). My shtpl uses a jinja-like syntax which, now that I use ansible a lot, I'm pretty familiar with:

$ cat /tmp/test
{{ aux=4 }}
{{ myarray=( a b c d ) }}
{{ A_RANDOM=$RANDOM }}
$A_RANDOM
{% if $(( $A_RANDOM%2 )) == 0 %}
$A_RANDOM is even
{% else %}
$A_RANDOM is odd
{% endif %}
{% if $(( $A_RANDOM%2 )) == 0 %}
{% for n in 1 2 3 $aux %}
\$myarray[$((n-1))]: ${myarray[$((n-1))]}
/etc/passwd field #$n: $(grep $USER /etc/passwd | cut -d: -f$n)
{% endfor %}
{% else %}
{% for n in {1..4} %}
\$myarray[$((n-1))]: ${myarray[$((n-1))]}
/etc/group field #$n: $(grep ^$USER /etc/group | cut -d: -f$n)
{% endfor %}
{% endif %}


$ ./shtpl < /tmp/test
6535
6535 is odd
$myarray[0]: a
/etc/group field #1: myusername
$myarray[1]: b
/etc/group field #2: x
$myarray[2]: c
/etc/group field #3: 1001
$myarray[3]: d
/etc/group field #4: 

More info on my github

0

Quick and dirty solution that safely supports full shell scripting.

template.sh

#!/bin/sh

EOF=EOF
exec cat <<EOF | sh
cat <<EOF
$(cat $1 | \
    sed 's|`|\\`|g' | \
    sed 's|\$|\\\$|g' | \
    sed "s|${OPEN:-<%}|\`eval echo |g" | \
    sed "s|${CLOSE:-%>}|\`|g")
$EOF
EOF

Simply run the following on any file you want to template.

sh template.sh file.txt.tmpl

file.txt.tmpl

uname: <% $(uname) %>
pwd: <% `pwd` %>
ls: $(ls)
df: `df`
SHELL: <% $SHELL %>
HOME: <% ${HOME} %>
USER: $USER
PATH: ${PATH}

The output would be something like the following.

uname: Linux
pwd: /home/clayrisser
ls: $(ls)
df: `df`
SHELL: /usr/bin/zsh
HOME: /home/clayrisser
USER: $USER
PATH: ${PATH}

If you want to write the templated file, simply redirect the output to a new file.

sh template.sh file.txt.tmpl > file.txt

You can change the open and close characters by setting OPEN and CLOSE.

file.txt.tmpl

uname: {{ $(uname) }}
pwd: {{ `pwd` }}
ls: $(ls)
df: `df`
SHELL: {{ $SHELL }}
HOME: {{ ${HOME} }}
USER: $USER
PATH: ${PATH}
OPEN="{{" CLOSE="}}" sh template.sh file.txt.tmpl

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .