3

I have several environments of the same application in different servers. The directory structure is very similar, so I'd like to use one single FileSet for all of them. For example:

- server1: /opt/env1/app/logs
- server1: /opt/env1/app/bin
- server1: /opt/env1/app/temp
- server1: /opt/env1/app/lib

- server2: /opt/env2/app/logs
- server2: /opt/env2/app/bin
- server2: /opt/env2/app/temp
- server2: /opt/env2/app/lib

- server2: /opt/env3/app/logs
- server2: /opt/env3/app/bin
- server2: /opt/env3/app/temp
- server2: /opt/env3/app/lib

I'd like to have something like:

FileSet {
  Name = "APP"
  Include {
    Options {
      signature = MD5
      Compression = GZIP
    }
    File = "/opt/${env}/app"
  }
  Exclude {
    File = "/opt/${env}/app/logs"
    File = "/opt/${env}/app/temp"
  }
}

Then:

Job {
  Name = "JOBENV1"
  #somehow set env variable as env1
  FileSet="APP"
  ...
}

Job {
  Name = "JOBENV2"
  #somehow set env variable as env2
  FileSet="APP"
  ...
}

Job {
  Name = "JOBENV3"
  #somehow set env variable as env3
  FileSet="APP"
  ...
}

2 Answers 2

2

After a few days with the solution, it seems to be working, so here it goes:

Up to the example, lets say we have the following jobs:

  • server1-env1-job
  • server2-env2-job
  • server2-env3-job

Then assuming the --job as a convention, the file set should look like:

FileSet {
  Name = "someapp-fileset"
  Include {
    Options {
      signature = MD5
      Compression = GZIP
    }
    File = "| bash -c \"echo %n | awk -F '-' '{print \$2}' | xargs -I ARG echo /opt/ARG\""
  }
  Exclude {
     File = "| bash -c \"echo %n | awk -F '-' '{print \$2}' | xargs -I ARG echo /opt/ARG/temp\""
     File = "| bash -c \"echo %n | awk -F '-' '{print \$2}' | xargs -I ARG echo /opt/ARG/logs\""
  }
}

For more information about the execution of commands in the file set look at the documentation

Full and incremental backups are running OK.

1

I don't believe you can do this - Bacula's configuration file is a configuration file, not a programming language or shell.

Your practical options are:

  1. Use identical directory structures on each machine
    This is the most practical solution from Bacula's standpoint. It requires the least additional work in the Bacula configuration file, and if your applications are "identical" it makes the most sense.

  2. Use a fileset per machine
    This is more annoying than option (1) in that you are now maintaining a bunch of nearly-identical filesets. It's most practical if you're generating your Bacula configuration with some other process that can automatically create the filesets.

  3. Use one fileset listing all the directories
    This works, but any time you add a new machine (and thus a new env# directory) you will be changing the global fileset and triggering full backups.

  4. Use a ClientRunBeforeJob directive to put the files in a uniform location
    This is a little dirty, but it works -- Have Bacula rsync the app environment to another location before it does its backup. (If you are on a Linux system you can also use mount --bind to put the app at a uniform location without affecting anything else.)
    The major downside of this approach is you double your storage requirements: you have the running production copy of the app, and the shadow copy Bacula makes in order to take its backup.

7
  • Thanks for the comments, about 1: I may have more than one environment in the same server, so it's not possible. About 4: I cannot afford to duplicate required storage (some apps have more than 500GB). I guess I'll have to choose between 2 and 3
    – ghm1014
    Jan 15, 2013 at 17:14
  • Given the restrictions (and your comment about app size) I would go with (2) - Triggering a full backup of ALL your systems when adding a new app environment would chew up a lot of storage and probably be cost/time prohibitive.
    – voretaq7
    Jan 15, 2013 at 17:16
  • @ghm1014 There's also the slim possibility I'm totally wrong and Bacula lets you do configuration variables now (but a quick scan of the docs says that's probably not the case) - you may want to hold off on accepting this answer in case someone comes up with a better one :)
    – voretaq7
    Jan 15, 2013 at 19:22
  • I may have found a way, it's a little bit tricky. I could use File = "| echo /opt/%n/app" Where %n is the name of the job that can be the name of the environment, also could use awk or any other bash tool to parse the jobName to obtain the environment (of course assuming an standard for the jobname that includes the env). @voretaq7 do you think that may work?
    – ghm1014
    Jan 16, 2013 at 19:02
  • @ghm1014 I think that will work (test to be sure), but I'm viscerally opposed to it because it looks scary :) Also I don't know how it will interact with the fileset-modified-so-do-a-full-backup logic, so definitely check that in a test environment - if it always runs a full that might be a Bad Thing
    – voretaq7
    Jan 16, 2013 at 22:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .