1

I need to build a system to store & maintain a huge amount (20 [TB]) of time-series data (for many different instruments), so that it would support the following requirements:

(1) fast appends of new data, as new data comes in
(2) fast retrievals of existing (already stored) data

There are 10,000 instruments and 1000 data fields (updated every 1-min) to save for each instrument. Once data is written to disk, it remains unchanged (no issues of concurrent writing/reading).

Since there would be no need for any joins whatsoever (typical query is: give me all instruments for field 'X' on interval 'Y'), I tend to store the data using flat binary files that will be named like this: fieldName.timeStamp.bin; this way, I would be able to store all the data in flat binary files (no need for a huge expense for a giant server / commercial database) and still, queries will be fast.

Since it's a lot of data (circa 20[TB]), I thought I would need some logic to distribute the files (fieldName.timeStamp.bin) between all of my machines. Here's what I had in mind: there will be a central machine to which all queries will be sent. this central machine (based on the field & timestamp requested) would route the query to the machine of interest, that would in turn return the requested data.

My questions are:

(1) is this design scalable as I think it is? any drawbacks?

(2) is there anything I am missing here that might hurt performance?

(3) is it really the best way to send all queries to a central machine, that would in turn route the query to the right machine? or would it be best to directly access the right machine (suppose I know which one it is ) using NFS?

(4) is there a faster way than NFS to access the right machine to read data from it? are there other methods for sharing all the data that on the data machines with client machines?

All of my machines use Ubuntu Linux. As can be understood, there will be many client machines that would access the various data machines and read (only read, not write) data from them. my goal is to have the data read as fast as possible.

Improve this question

3 Answers 3

Reset to default
2

You may also want to take a look at OpenTSDB, a Hadoop-based system for storing and retrieving massive time-series data. I've never used it, but it sounds handy and at least near your purposes.

Improve this answer
13
  • Thank you. Does my design makes sense, at least?
    – user76976
    Apr 3, 2011 at 22:34
  • Things to think about: will you ever be deleting or archiving data? If so, how do you support that? How many clients are you talking about? If there are relatively few clients, then at any given time a number of your data machines may be idle when they could have been serving data if you distributed the workload more evenly. Do you expect your clients to parse the .bin files, or would it give you better performance to have your data machines do it and only send over the query results? I suspect the latter is correct, in which case you should avoid NFS.
    – justarobert
    Apr 3, 2011 at 23:12
  • My comments about performance are unlikely to be that big of a deal if you're looking for bandwidth and latency to support a seek and 500KB read in 1s, though. At that point, simplicity and robustness are more likely to matter. You may want to consider storing multiple copies of the data on separate machines. What happens in your design if one data machine goes offline?
    – justarobert
    Apr 3, 2011 at 23:18
  • @justarobert: There very well might be 100s of clients. Here's what I have in mind though: for every single data machine, there will be 3 machines that mirror it (meaning, a cron rsync will make 3 copies of every data machine). When a read request is sent, the central machine will randomize one machine out of the four (1 data machine + 3 copies = 4 machines) and use it to retrieve the requested data. if / when a data machine gets corrupted or there are some issues, one of the mirrored machines will replace it. Does it make sense?
    – user76976
    Apr 3, 2011 at 23:33
  • @justarobert: I would of course be happy with MUCH BETTER PERFORMANCE that 500KB read in 1sec. Yet, just to make this clear, what I meant was that I want a system that would support say 200 clients each (concurrently) being able to read 500KB in 1sec (at least). Do you see any issues using NFS for this? are there better / more robust ways?
    – user76976
    Apr 3, 2011 at 23:37
2

Tahoe Least Authority File System may solve many of these problems automatically, especially if you can work with their tools for retrieving the data. At least, I'd give it a look before making my own system. Without data on what the real bandwidth and latency requirements are, I can't say much more.

Improve this answer
3
  • thank you. can you give me some pointers how to estimate the real bandwidth and latency requirements? Would it help if I say that I'd like every client machine to be able to read any 500KB file in one of the data machines in less than 1[sec]?
    – user76976
    Apr 3, 2011 at 22:31
  • any feedback on my design? does it at all make sense? am I in the right direction?
    – user76976
    Apr 3, 2011 at 22:35
  • I don't see anything wrong with the design, but I don't have any real experience with it myself (I have one 10TB file server at work, and it just offers NFS, Samba, and Appletalk to various clients). At this point, I'd start check with Tahoe's mailing lists and get their opinions. I think 500KB in 1 second would be easy to meet, though.
    – Mike Renfro
    Apr 3, 2011 at 22:50
1

A few notes:

1) Using a centralized server seems unnecessary here. Why not make a hash of the filename and use a simple sort to decide which server to store / get the files? That way you don't need a central server to store / write the files.

2) Given the scale of the system you are talking about, I would look into using Lustre or GLuster to do the file system stuff for you instead of using NFS. Let them do the hard work for you. Both are used for systems much larger then this and have a solid track record.

3) If you do decide to role your own setup, I would take a strong look at OpenSolaris / Nexenta w/ZFS. For file systems that large, some of ZFS's strengths become really helpful:

a) ZFS does intelegent raid rebuilds. I can rebuild 16tb of data on a 10x2tb raid 50 drive setup in 30 hours. Which is much faster then if I was doing the same kind of rebuild with a hardware raid card. b) ZFS does not need to fsck, even with ext3/4 the fsck on partitions that large is going to be really painful. c) ZFS's I/O scheduler for writes is very strong. You can add a single SSD to hold the ZIL logs / LARC2 cache and get most of the gains of a SSD based storage system with the large data retention of hard drives. d) ZFS has a very robust NFSv4 server built in. Sharing is easy to configure. e) ZFS has built in file system level de-duplication which might be a huge win for you if the instrument readings often return similar results.

Improve this answer
1
  • thank you. Regarding (1), you are right. The problem is, if I don't 'send' the WRITES through a 'central' machine, then in order to be able to read/write FAST, I will need every machine to have an open SSH connection to all other machines at all times. is it a porblem? also, I am not familiar with the other filesystems; would it be so bad to use ext4 if every machine only stores say 500[GB] of data on it? would I still need ZFS or Lustre? (are they hard to install / maintain) ?
    – user76976
    Apr 4, 2011 at 15:19

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .