0

Short version

I am looking for a glorified cron manager with a web interface to manage backups and uploading to cloud storage. (and yes webmin has a pretty good cron scheduler plugin)

Long version

In the old days enterprise backup systems were behemoth monstrosities costing 6 or more figure dollar prices or roll your own low level tools that are tightly coupled to their tape archiving forebears.

Examples of the former are HP data protector and IBM Tape storage systems, and of the latter I am thinking of Bacula and amanda.

These were great systems for their time, and did much of the grunt work for you ie.

  1. scheduling
  2. reporting
  3. alerting
  4. managing media
  5. and providing a very old school GUI (I think I remember Hp data protector had an ActiveX component, who uses activeX?)

However recent changes are away from TAPE and local storage and towards cloud or network based backup storage locations.

In addition, other trends such as automated provisioning has meant that there is far less file and directory backup "stuff"* and much more database and binary object backups objects such as SQL databases and SVN, or other repositories.

Another observation is that It costs money to leave data in the cloud

hence it is necessary to have some easy to use management interface to archive old backups on S3 or openstack swift networks.

So I end up in the situation where most of the backup management tools I am familiar with are not suitable for my backup purpose and now most of my apps need some sort of database backup dump, and the uploading the backed up files to network or cloud storage needs to be managed as well as the jobs.

For mySQL backups, I have ended up with a stack of software which is something like this;
1. percona xtrabackup to generate full and incremental
2. cron to initiate the backup jobs
3. duplicity with python-cloudfiles to sync the backup jobs to cloud
4. nagios and logwatch to monitor and alert for failures.

However this all seems a bit crufty.

The duplicity/cloudfiles solution is remarkably easy to configure with rsync like semantics, and xtra seems to do mostly hot backups without a hitch, but none of this really inspires the confidence that the One-stop-shop software like HP data protector, or tivoli did.

What I would like to do is tie all this together using some generalized job scheduling manager that could run the jobs, report on the success, and could handle scheduling of tasks for ageing old backups.

I am aware of some Java job scheduling tools like Quartz, but that is really too low level for my requirement, really I am thinking of some sort of glorified cron with reporting.

Updates

mysql-ZRM is interesting, it supports scheduling and basic reporting, and also there is an xtrabackup plug-in. however it really lacks a web interface and is probably not quite up to the percona level as it apparently not really maintained any more.

xtrabckup manager is good, supports scheduling. However it only supports native xtrabackup incrementals, which are massively large if you have lots of myisam tables.

holland and zmanada are a couple of derivative or commercial projects I am keeping an eye on.

duplicity and python-cloudfiles is quite the bllcks at the moment, so that is what I am using to sync the backup files to a openstack swift storage.

3
  • While I appreciate you want something 'prettier' than some cron jobs, it really is ideal if you pull it all together using Puppet/Chef. I don't think you're going to find an all-in-one(wonder) system. If you do, it's probably going to be built in house. Best of luck though!
    – Publiccert
    Feb 20, 2012 at 18:13
  • yep, we have chef. Automated deployment and configuration has made much of the OS and package level backups pointless, as you can just as easily re-run the recipe. (and that includes installing from source, as that is wrapped in a ruby puppet/chef recipe) That is kind of what I am getting at, is that much of the "enterprise backup" stuff is no longer fit for the purpose.
    – Tom
    Feb 20, 2012 at 18:17
  • You're completely right. There aren't very many GOOD enterprise level options. I'm working in an environment where my team alone handles 6000+ servers. My team is myself and 8 sysadmins. Forget about the other 120+ admins and all THEIR servers. My point being...enterprise level options suck. We use TSM for whatever that's worth.
    – Publiccert
    Feb 20, 2012 at 20:04

3 Answers 3

1

Have you look at the Open Source Job Scheduler yet?

1
  • Not yet, it definitely looks flexible enough for my purposes, after quick read of the docs it looks like it might be a tad low level but its my best bet at the moment.
    – Tom
    Feb 21, 2012 at 8:40
1

You might take a look at XtraBackup Manager, by Lachlan Mulcahy. (It's not a Percona product; it's a community-oriented manager for Percona XtraBackup). If it doesn't meet your needs now, it might meet some or all of them in the future.

2
  • Are you stalking me Baron!? i see you answering my questions on all these different forums, maybe you should just give me your email address and cut out the middle man ;-) Only joking, thanks for that answer on the percona forums I only discovered percona this last week and they are already my favourite tools.
    – Tom
    Feb 21, 2012 at 8:35
  • I will download it and give it a try today, it looks like the sort of thing that is going in the right direction. thanx
    – Tom
    Feb 21, 2012 at 8:41
0

The answer to this question, might actually be hudson/jenkins, because it actually provides all the scheduling and reporting tools, and handles alerts and conditionals and can wrap right around stuff like percona.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .