32

(Related to Callbacks or hooks, and reusable series of tasks, in Ansible roles):

Is there any better way to append to a list or add a key to a dictionary in Ansible than (ab)using a jina2 template expression?

I know you can do something like:

- name: this is a hack
  shell: echo "{% originalvar.append('x') %}New value of originalvar is {{originalvar}}"

but is there really no sort of meta task or helper to do this?

It feels fragile, seems to be undocumented, and relies on lots of assumptions about how variables work in Ansible.

My use case is multiple roles (database server extensions) that each need to supply some configuration to a base role (the database server). It's not as simple as appending a line to the db server config file; each change applies to the same line, e.g. the extensions bdr and pg_stat_statements must both appear on a target line:

shared_preload_libaries = 'bdr, pg_stat_statements'

Is the Ansible way to do this to just process the config file multiple times (once per extension) with a regexp that extracts the current value, parses it, and then rewrites it? If so, how do you make that idempotent across multiple runs?

What if the config is harder than this to parse and it's not as simple as appending another comma-separated value? Think XML config files.

  • Y'know what? I like the cut of your side-effect-ful jib ☺ – DomQ Aug 1 '18 at 13:49
13

You can merge two lists in a variable with +. Say you have a group_vars file with this content:

---
# group_vars/all
pgsql_extensions:
  - ext1
  - ext2
  - ext3

And it's used in a template pgsql.conf.j2 like:

# {{ ansible_managed }}
pgsql_extensions={% for item in pgsql_extensions %}{{ item }}, {% endfor %}

You can then append extensions to the testing database servers like this:

---
# group_vars/testing_db
append_exts:
  - ext4
  - ext5
pgsql_extensions: "{{ pgsql_extensions + append_exts }}"

When the role is run in any of the testing servers, the aditional extensions will be added.

I'm not sure this works for dictionaries as well, and also be careful with spaces and leaving a dangling comma at the end of the line.

  • You can, but you have to do it all in group_vars, the roles can't take care of the details of setting up the extensions themselves. It's appending vars from roles that I'm particularly looking for, so one role can append to a var exposed by another role. – Craig Ringer Jul 14 '15 at 6:10
  • Does your base role know about each extension role? I've had a similar case where I was able to leave the concatenation up to a with_items sentence. – GnP Jul 14 '15 at 15:39
  • no, and that is really the issue. In one deployment the base role might be al – Craig Ringer Jul 14 '15 at 23:16
  • 4
    It seems like if you try to do this to concatenate two lists, it thinks it's an infinitely recursive template because the left hand side is also on the right hand side. Am I misunderstanding how to use this? – Ibrahim Mar 24 '16 at 23:36
  • 1
    @spectras At least as of Ansible 2.7 this does NOT work. As Ibrahim suggested, this causes an error: "recursive loop detected in template string". – rluba Oct 25 '18 at 11:08
32

Since Ansible v2.x you can do these:

# use case I: appending to LIST variable:

      - name: my appender
        set_fact:
          my_list_var: '{{my_list_var + new_items_list}}'

# use case II: appending to LIST variable one by one:

      - name: my appender
        set_fact:
          my_list_var: '{{my_list_var + [item]}}'
        with_items: '{{my_new_items|list}}'

# use case III: appending more keys DICT variable in a "batch":

      - name: my appender
        set_fact:
          my_dict_var: '{{my_dict_var|combine(my_new_keys_in_a_dict)}}'

# use case IV: appending keys DICT variable one by one from tuples
      - name: setup list of tuples (for 2.4.x and up
        set_fact:
          lot: >
            [('key1', 'value1',), ('key2', 'value2',), ..., ('keyN', 'valueN',)],
      - name: my appender
        set_fact:
          my_dict_var: '{{my_dict_var|combine({item[0]: item[1]})}}'
        with_items: '{{lot}}'
# use case V: appending keys DICT variable one by one from list of dicts (thanks to @ssc)

  - name: add new key / value pairs to dict
    set_fact:
      my_dict_var: "{{ my_dict_var | combine({item.key: item.value}) }}"
    with_items:
    - { key: 'key01', value: 'value 01' }
    - { key: 'key02', value: 'value 03' }
    - { key: 'key03', value: 'value 04' }

all the above is documented in: playbooks_filters

  • 1
    use case IV just appends u'(': u\"'\"}" – ssc Feb 25 '18 at 12:50
  • 1
    thanks, @ssc. I noticed it's not working with ansible 2.4.x (FIXED) – Max Kovgan Feb 26 '18 at 14:50
  • according to usecase #4, I have added the default value to handle the undefined error in my scenario: set_fact: my_dict_var: '{{my_dict_var|default({})|combine({item[0]: item[1]})}}'. The undefined error comes when some filtering is used or no results is registered. – S.K. Venkat Aug 7 '18 at 7:10
3

you need to split the loop into 2

--- 
- hosts: localhost
  tasks: 
    - include_vars: stacks
    - set_facts: roles={{stacks.Roles | split(' ')}}
    - include: addhost.yml
      with_items: "{{roles}}"

and addhost.yml

- set_facts: groupname={{item}}
- set_facts: ips={{stacks[item]|split(' ')}}
- local_action: add_host hostname={{item}} groupname={{groupname}}
  with_items: {{ips}}
0

Not sure when they added this, but at least for dictionaries/hashes (NOT lists/arrays), you can set the variable hash_behaviour, like so: hash_behaviour = merge in your ansible.cfg.

Took me quite a few hours to accidentally stumble upon this setting :S

  • this is very handy, but beware to enable it e2e on existing codebase. may break some eggs. – Max Kovgan Aug 22 '18 at 15:28
-4

Ansible is an automation system, and, concerning configuration file management, it's not very different from apt. The reason more and more software offers the feature to read configuration snippets from a conf.d directory is to enable such automation systems to have different packages/roles add configuration to the software. I believe that it is not the philosophy of Ansible to do what you have in mind, but instead to use the conf.d trick. If the software being configured does not offer this functionality, you may be in trouble.

Since you mention XML configuration files, I take the opportunity to do some whining. There's a reason for the Unix tradition of using plain text configuration files. Binary configuration files do not lend themselves well to system automation, so any kind of binary format will give you trouble and will likely require you to create a program to handle the configuration. (If anyone thinks XML is a plain text format, they should go have their brains examined.)

Now, on your specific PostgreSQL problem. PostgreSQL does support the conf.d trick. First, I would check whether shared_preload_libraries can be specified multiple times. I did not find any hint in the documentation that it can, but I would still try it. If it cannot be specified multiple times, I would explain my problem to the PostgreSQL guys in case they have ideas; this is a PostgreSQL issue and not an Ansible issue. If there is no solution and I really couldn't merge the different roles into one, I'd implement a system to compile the configuration on the managed host. In this case, I'd probably create a script /usr/local/sbin/update_postgresql_config which would compile /etc/postgresql/postgresql.conf.jinja into /etc/postgresql/9.x/main/postgresql.conf. The script would read the shared preload libraries from /etc/postgresql/shared_preload_libraries.txt, one library per line, and provide them to jinja.

It is not uncommon for automation systems to do this. An example is the Debian exim4 package.

  • PostgreSQL supports a conf.d include mechanism and thankfully uses plaintext files. However, there are some configuration options where multiple extensions may have opinions about it - e.g "increase max_wal_senders by 10 from whatever it was before". – Craig Ringer Apr 28 '15 at 7:05
  • 4
    It seems like you're saying that the application should be changed to work around limitations in the configuration management system, or I should give up on having re-usable roles. – Craig Ringer Apr 30 '15 at 4:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.