3

I've got some data that is being backed up to tape via bacula, and at some point soon I am going to need to migrate to non-tape storage.

Bacula supports to-disk archiving, but it occurred to me that if only there was some virtual tape library interface to S3 or openstack Swift, or some other object store service, I could just swap out the configurations and maintain the pre-existing archiving, ageing and scheduling configurations seamlessly.

I had a google, and I can see that Amanda (using a Device API) and a bunch of backup products have S3 object store back-ends, but I don't see any projects that would drop-in replace using *nix /dev/st0 device interface semantics.

This may well be because I have no idea, and that the idea is preposterous and impossible, etc, however I'd be interested to know if I have missed any obvious things before I delve into some hacked up replacement.

So clarify the the platform options - its not hosted on AWS and there is at present no EBS available and the target is an in-house S3 compatible openStack Swift object store, so bandwidth and storage costs are internal.

3 Answers 3

1

This is more than a questionable approach for anything anyone could consider a backup but you should be able to give it a try. The first thing you need would be anything that presents a block interface on top of S3. Something like a NBD server with a S3 backend - have a look at this.

After you got that working you need something that presents a file on top of a block device and offers a SCSI target/virtual HBA to your system with a changer and a tape drive. Have a look at this project here.

See the S3NBD project (first link) for additional rationale why running your backup and recovery with S3 might be a bad idea (with the current S3 pricing scheme).

2
  • The economics of backup to S3 are not an issue here because the S3/swift target in this case is going to be an in-house OpenStack infrastructure.
    – Tom
    Jan 19, 2012 at 2:04
  • The second one looks promising as a fun to try project, so I will check it out. The first project has been mothballed...
    – Tom
    Jan 19, 2012 at 2:25
1

The bad thing about emulating a tape on EC2 is that in order to emulate a tape, you must emulate all the aspects. That includes the idea that when you load a tape and aren't planning to write at the start, you must seek through all that data.

I believe a simpler course of action would be to interface EBS volumes as block devices in Bacula.

What might be an even simpler course of action would be to use Media Type = Fifo and link that to a program that writes out to EBS. For restores, you'll have to work something else out, but it'll get your data moved.

2
  • I will edit the question to make it more clear, but the target is actually an S3 compatible OpenStack swift object store that is being setup internally in our Comp, and EBS is not available at the moment, hence S3 or swift API is necessary.
    – Tom
    Jan 19, 2012 at 2:06
  • @TomH One can interface S3 objects to FIFO pipes as well... even more easily, I'd say. Jan 19, 2012 at 16:24
1

You might try MHVTL it emulates a virtual tape library and uses files/directories as storage. You could S3FS or other mechanism to emulate files/directories on which MHVTL could sit. You also could modify MHTL to just move its files to/from S3/Swift like target on mount/unmount. It looks like /dev/tapedevice and a scsi media changer, so backup products might work unchanged. Dont know about support.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .