Is it realistic to setup a 100 TB database (about 90 TB actually) on PostgreSQL without data sharding between a number of nodes? Are there any success stories / examples about similar setups?
migrated from stackoverflow.com Mar 29 '11 at 6:46
This question came from our site for professional and enthusiast programmers.
50K writes per second that need to be absorbed is more than a challenge usually. Even in synthetic benchmarks with quite simple inserts, pgsql's limits tend to max out around roughly 10K/s - and there you don't even have such a large beast in terms of db size.
Also the IO system for that single pgsql node is going to be interesting as even with RAID10 and asuming that 50K inserts are going to be equal to just 50K IOPS (which is probably wrong - but depends on your db scheme and indices), you are going to need roughly a hundred disks paired with a very good array that saves you from buying several hundred disks to service those writes in a timely manner.
If sharding is easy and you expect such a huge write load then go for sharding. Writes can be very difficult to scale.
I apologize for my bluntness
If you have the money or even the need for 90TB of data storage then this is exactly where you SHOULDN'T ask. There are lots of companies out there that sell this kind of product and skills ( EMC ) and for that you actually get a solid product. You are kidding yourself if you think this is an area you can save some money by doing yourself. especially if you have to ask the community.
This isn't something you try to save money on, this is something you get right the FIRST time. go call HP, Dell, EMC ect.. and ask them about this, they will be more then happy to give you PROFESSIONAL advice and also sell a product. If something like this breaks, you or the company probably wont get a 2nd chance.
A company can go bankrupt from a single broken tape backup drive, imagine the cost of 90TB of data being lost! that's not something you want to live with.
Get a professional product and if you don't think its something you can afford then i would seriously reconsider your companies priorities.