The oddest thing is happening during this Puppet run (using Puppet Apply) and it has me perplexed. I've tried my usual channels to find solutions but I cannot find anything on the topic. I apologize for being so verbose, I just don't want to leave anything out :\


My Class

class c2c::profile::app::logio::stage_support {

  # Log.io plugin
  # Create plugin directories
  exec { "create_codec_dir":
    command => "/bin/mkdir --parents /etc/logstash/plugins/logstash/codecs --mode=0775",
    creates => '/etc/logstash/plugins/logstash/codecs',

  # Install plugin
  file { "logio_plugin_file":

    # update: added in response to a serverfault comment
    ensure  => file,

    path    => '/etc/logstash/plugins/logstash/codecs/logio.rb',
    mode    => '0775',
    owner   => 'root',
    group   => 'root',
    source  => 'puppet:///modules/c2c/logstash/logio_codec.rb',
    require => Exec['create_codec_dir'],
    notify  => Service['logstash'],


.. and my Puppet run is skipping the logio_plugin_file instruction (or, for whatever reason, is not copying the file).


Logs, Etc

The debug log is rather large, and did not want to paste the whole thing, so here is every line in the Puppet run that mentions 'logio_plugin_file', but I'll gladly provide more if needed.

Debug: /File[logio_plugin_file]/seluser: Found seluser default 'system_u' for /etc/logstash/plugins/logstash/codecs/logio.rb
Debug: /File[logio_plugin_file]/selrole: Found selrole default 'object_r' for /etc/logstash/plugins/logstash/codecs/logio.rb
Debug: /File[logio_plugin_file]/seltype: Found seltype default 'etc_t' for /etc/logstash/plugins/logstash/codecs/logio.rb
Debug: /File[logio_plugin_file]/selrange: Found selrange default 's0' for /etc/logstash/plugins/logstash/codecs/logio.rb
Debug: /File[logio_plugin_file]/require: requires Exec[create_codec_dir]
Debug: /File[logio_plugin_file]/notify: subscribes to Service[logstash]
Debug: /File[logio_plugin_file]: Autorequiring File[logstash_codec_plugins]


The term 'logio_plugin_file' is only mentioned as part of two Puppet::Util::Log entries, and no File[logio_plugin_file] entry exists.

Below is a quick snippet of those two entries:

- !ruby/object:Puppet::Util::Log
  level: !ruby/sym debug
  time: 2013-12-06 17:30:38.009095 +00:00
    - debug
    - file
    - logio_plugin_file
    - class
    ... (25 more ) ...
  line: 41
  source: /File[logio_plugin_file]/require
  file: /tmp/vagrant-puppet/modules-0/c2c/manifests/profile/app/logio/stage_support.pp
  message: "requires Exec[create_codec_dir]"

  .. further down, mostly the same except ..

  message: "subscribes to Service[logstash]"



This file is included within a large catalog, with dozens or hundreds of other classes, but what's weird is if I include this class directly using a quick testing manifest (also using puppet apply), it works like a charm. (Although I did have to remove the service requirement)

node default {
  class { 'c2c::profile::app::logio::stage_support': }

.. and I get this in my output ..

Notice: /File[logio_plugin_file]/ensure: defined content as '{md5}41d00952843b8159b95ce4fcd8015cda'

.. and this in last_run_report.yaml ..

File[logio_plugin_file]: !ruby/object:Puppet::Resource::Status
  resource: File[logio_plugin_file]
  file: /tmp/vagrant-puppet/modules-0/c2c/manifests/profile/app/logio/stage_support.pp
  line: 40
  evaluation_time: 0.023071
  change_count: 1
  out_of_sync_count: 1


Additional Points, Info, and Tests

  • I only just added the Notify => Service['logstash'], and this behavior was occuring before I did so.
  • No errors are emitted
  • I added a few notify{} calls in my stage_support class to ensure it was being properly included in the catalog, and it is.
  • Exec['create_codec_dir'] is creating my directory
  • I tried changing the path param to logio2.rb to see if the class
  • I tried requiring File['logio_plugin_file'] from another resource. The requiring resource ran, but the file was not created.
  • Update: I tried reducing my file type call to only include the title plus ensure, path, and source, which had no effect.
  • Update: I tried renaming the resource, (e.g. logio_plugin_file_x), which did not help.

    [root@dev ~]# puppet --version
    [root@dev ~]# facter --version
    [root@dev ~]# cat /etc/redhat-release
    CentOS release 6.4 (Final)


Important Note

This problem comes on the tail end of me doing a code restructuring, so that's almost certainly to blame, but I am not able to troubleshoot this.

Any help is much appreciated!

  • 1
    Is that resource included in the run report's resource list (/var/lib/puppet/state/last_run_report.yaml)? – Shane Madden Dec 6 '13 at 1:39
  • It is mentioned twice, yes, as the 'source' and one of the 'tags' for two !ruby/object:Puppet::Util::Log entries. Those messages are: "requires Exec[create_codec_dir]" and "Autorequiring File[logstash_codec_plugins]". Otherwise, a search for logio_plugin_file does not return any additional results within that file. – LukeChavers Dec 6 '13 at 15:21
  • When running puppet apply with my testing manifest (as mentioned as the "strangeness" above), there is an additional last_run_report.yaml entry that does not exist in the standard run: File[logio_plugin_file]: !ruby/object:Puppet::Resource::Status .. I'm guessing that is what you were looking for in your question. – LukeChavers Dec 6 '13 at 15:26
  • That's.. very strange. Try adding ensure => file for paranoia's sake? And I think it is going to be, but verify the resource is in the catalog file at /var/lib/puppet/client_data/catalog/hostname? – Shane Madden Dec 6 '13 at 17:15
  • I added ensure => file to no effect. I thought you might have nailed it there and perhaps something had overrode the default action. I checked for the file also, but my /var/lib/puppet/client_data directory is empty, might be because I am using apply instead of agent, I'm not sure. I'm tempted to rebuild the VM from scratch, but I worry that I might sweep the problem under the rug and have to fight it again later on if I do. – LukeChavers Dec 6 '13 at 17:42

As expected, the source of the problem turned out to be simple, though I still do not understand why Puppet did not fire an error about this:

profile/logstash.pp: (elsewhere)

file { 'logstash_plugin_sub':
  path      => '/etc/logstash/plugins/logstash',
  ensure    => 'directory',
  owner     => 'root',
  group     => 'root',
  mode      => '0775',
  require   => [ File['logstash_plugin_sub'] ]

Note the circular reference in the require: File['logstash_plugin_sub'] -> File['logstash_plugin_sub']

The resulting behavior is very weird, and I found the problem using this test:

file { "logio_plugin_file_a":
  path    => '/etc/logstash/logio.rb',
  source  => 'puppet:///modules/c2c/logstash/logio_codec.rb',
file { "logio_plugin_file_b":
  path    => '/etc/logstash/plugins/logio.rb',
  source  => 'puppet:///modules/c2c/logstash/logio_codec.rb',
file { "logio_plugin_file_c":
  path    => '/etc/logstash/plugins/logstash/logio.rb',
  source  => 'puppet:///modules/c2c/logstash/logio_codec.rb',
file { "logio_plugin_file_d":
  path    => '/etc/logstash/plugins/logstash/codecs/logio.rb',
  source  => 'puppet:///modules/c2c/logstash/logio_codec.rb',

The idea being, principally, try adding my file at deeper and deeper locations in the tree. This idea came to me as a result of being able to change the path in my original example, and achieve success.

So, the following was created:


But the two deeper resources were skipped, so I knew my problem centered around /etc/logstash/plugins/logstash. Upon close inspection, I found the circular reference.

Hope this helps someone, and thank you to Shane Madden for his time.

  • Wow, good find - it's crazy that it's not throwing a big angry error message on that! That seems like a bug, might be worth reporting to them? – Shane Madden Dec 10 '13 at 2:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.