(Open Source) Cloud-Filesystem to run a Database on Top?

Question

what are current "technologies and implementations" to get a filesystem with unlimited capacity by using single servers with their harddisk to form a "grid/cloud filesystem"?

I need to have unlimited space (by adding further servers) but it must be a filesysem that is capable of running a database on top.

I know of Apache Hadoop but that seems not be be Ideal for running a DB on top of it (or am I wrong??)

And iSCSI seems to be "remote/networked" but I do not know how and if this is clusterable?

thank you very much!! jens

Answer 1

Two words: STUPID IDEA. This is like asking "what is the best way to win in formula one with a 40 ton truck". The design principles between distributed file systems and database storage systems are orthogonal - they go for totally different targets. Actually most proper databases will try living in a handfull of files - so they end up in one (not controlled) node anyway (per file).

For a database you want defined IO performance, seriously optimized latency - you do a lot of very time critial IO. Actually larger database storage systems are designed around IOPS - IO operations per second - optimizing delays out. Storage SIZE Is normally not critical - in the pre SSD days you bought discs not for space primarily, but for IO performance. I once had a chance to work with a database distribute over 190 discs in a SAN - because it needed the hugh IO performance. The discs were not particularly full.

For a distributed file system you want ease of management, transparenty of location, focus on storage size instead of defined IO performance. You actually can not guarantee IO performance on a good enough level as you get a very unreliable (as in: can change) infrastructure below. You rely on caches to handle performance in many parts, which is redundant with what the database does itself and will simply not work in a proper optimized database scenario.

One is a 40 ton truck for moving lots of things, the other is a highly tuned formula one car. You will not ever get them into the same boat because they are designed based on different assumptions.

Answer 2

Unless you have great bandwidth between the nodes, I don't think this is going to fly.

You could set up a distributed block device with drbd, for instance, and run a RAID setup over several boxes, mount the fs on a single node and run your database server. But the performance is going to suck unless you have LAN-level communications performance.

What are you storing in your database which does not fit a single server? Is it really cheaper for you to buy multiple boxes vs. one big server?

Have you looked into sharding?

Are you storing files in your database? If you are, could you split those off?

Answer 3

This depends on the type of database you're talking about.

If you're looking for a SQL-based database like MySQL or PostgreSQL, you're barking up the wrong tree (with a couple of interesting exceptions, see below).

If you're trying to run something like a glorified key-value store like Cassandra or HBase, read on. When the partitioning algorithm is order-preserving, you can do things similar to what you can do in SQL, except you have to do joins manually and you get the advantage of being able to map-reduce for aggregates and more complex analysis tasks (depending on the DB platform), potentially with the help of projects like Pig and Hive.

For Cassandra, you don't need a specific file system. Again, if you need to run queries and have things similar to indexes, you'll want to use an order-preserving partitioning algorithm.

For HBase, you need to use HDFS as the base FS. HyperTable used to work under either HDFS or KFS; right now, I don't see any mentions of KFS but I also saw something about being able to work in a standalone fashion like Cassandra. I'm not too sure about partitioning and sorting on HyperTable, but I know that HBase by default stores records in order and supports indexes.

Now the interesting exceptions.

There is a project called Hadoop DB which integrates PostgreSQL with Hadoop and may, depending on what you're trying to do, serve your needs.

There's also the crazy idea of writing a storage engine for MySQL that interfaces with one of the systems above. The joins and aggregates and such would be done for you on the MySQL side while the data storage, indexing and retrieval would be your job. You'd also need to coordinate for transactions if you choose to support them. This means you could have multiple load-balanced MySQL servers talking to a huge HBase cluster. The only thing you'd be missing is the ability to push out map-reduce tasks to make aggregates and analysis over large parts of your data set more efficient. But you could do that periodically outside MySQL, and store the analysis results in another table that you can query with MySQL, again, depending on your needs.

Answer 4

A relatively simple way to do this is create a virtual machine install at a provider like Linode. They provide pre-installed versions of most Linux distributions.

Manage your storage under LVM. If you need more storage, the provider can add it for you - just have LVM add the additional volume to the volume group.

Which filesystem you use is a separate question.