How do you configure and work with your Puppet filebucket?

I'd like to:

  1. Store file bucket data on a single server
  2. Be able to audit/parse/prune the file bucket when necessary
  3. Ensure that filebucket data is securely transferred between client->master

Well, in a class that's included by all my nodes, I've got:

filebucket { puppet: server => "puppet.example.edu" }

The default in the File type is to backup to a local filebucket named "puppet". By changing the "puppet" filebucket to a server filebucket, you get server-based filebucket by default.

Alternately, if you want to preserve the option of overriding one specific file to use a local filebucket, you could do:

filebucket { main: server => "puppet.example.edu"; }
File { backup => main }

See http://docs.puppetlabs.com/references/latest/type.html#filebucket for more details on options.

This accomplishes item #1 because it tells the nodes to all use the same single server for the filebucket. Item #3 comes along for free because it's still all going over an SSL-based connection with SSL-certificate verification.

Filebucket is mostly useful in case of recovery, which is likely to be the same day. In that case, look at the report and use the "filebucket" or "puppet filebucket" command to retrieve the original content based on the md5sum in the report.

Item #2 is where things gets tricky...

I prune it with a script like this:

find /var/lib/puppet/clientbucket/ -type f -mtime +45 -atime +45 -delete

That removes anything that's older than 45 days and hasn't been accessed at all in that time. The 45 days is based on our backup and backup retention policy, since it's long enough for a backup with a long retention to have happened and give us a theoretical 18 month recovery time.

What kind of parsing are you looking for? The bucket setup on the server is a hierarchy organized by md5sum, and inside a directory name matching the md5sum, there's "paths" to tell you which file and "contents" is the actual file. You need to look at the reports to see what system it came from.

I don't do any auditing. What kind of auditing are you looking for? That could mean many things.

| improve this answer | |
  • Thanks, actually this is quite helpful. I didn't realize that configuring a centralized filebucket server at the puppetmaster was that simple! There are some feature requests tickets open for better filebucket management tools @puppetlabs. Seems like it's just a bit immature at the moment. My central filebucket is now filling up with files, but I don't seem to have a way to determine which host the files originated. Your 1-liner is helpful however. – robbyt Feb 28 '11 at 3:43
  • I think if multiple hosts have identical files go into the filebucket, it'll only get stored once, so the relationship between files in the filebucket and hosts isn't simple. – freiheit Feb 28 '11 at 17:11

One suggestion would be to add to files to a revision control server (svn,git). I store all the files that correlate with a specific module in the modules directory under /files. When a file is modified its checked into SVN and pushed to the master via hooks.

For example, I have a module that manages postfix and pushes the /etc/postfix/virtual files to clients. That files is stored under /etc/puppet/modules/postfix/files.

This gives you the advantage of reusing code that is created in your modules as well as organizing files per modules.

| improve this answer | |
  • 2
    filebucket != fileserver. I made the same mistake reading the question the first time. – Sirex Feb 25 '11 at 12:08
  • yes, I'm talking about the puppet filebucket. The system in Puppet that relocates replaced files into a staging area instead of just deleting them. linux.die.net/man/8/filebucket – robbyt Feb 26 '11 at 22:28
  • Actually I don't understand it either. If you have all versions of config files in git repo, why mess with filebucket? Accidently deleting someting else? Deleting something else is pretty explicit in puppet. IMHO filebucket was designed for folks without version control at all. – Art Shayderov Feb 28 '11 at 12:35
  • No, filebuckets are only useful for manual retrieval of accidentally removed/overwritten files (e.g., you look in the log for the md5 sum and retrieve the file with that sum from the filebucket), but when transactions are fully supported filebuckets will be used to undo transactions. – Martijn Heemels Jul 15 '11 at 9:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.