-5

I do algorithmic trading as a hobby, and I'm about to step up my game in terms of latency.

I have two questions regarding appropriate hardware and database software.

Question 1.

Before I will purchase my hardware, I would like to know what is in general a good setup for storing many datapoints in multiple tables.

The current CPU and harddrive components I'm contemplating are roughly the following:

  • Dual CPU: Dual Intel XEON E5-2660 v2 Deca-Core 2.2GHz (3.0GHz Turbo) 25MB Cache
  • Dual Seagate 4TB SSHD
  • Kingston ECC 1600MHz (8x8 GB)

Would a double CPU + double SSHD be sufficient to be able to process many datapoints (1.000.000+) per second?

And especially, I am extremely curious to the following:

  • Is there any advantage in having more harddrives?

  • What if I split my database tables over multiple harddrives, would that increase speed?

  • Are there any other hardware (especially harddrive) setups that could increase speed when working with databases?

Question 2.

What would be the best database software I can run to store this many datapoints as fast as possible?

Also, which database software allows for simultaneous reading+writing to the same table? (so no locks)

I know of a database system called 'MemSQL' which seems appropriate. Does anybody have any experience with MemsQL? Do you know any other low-latency database software that is recommendable?

I look forward to your replies. Thanks in advance.

closed as off-topic by Sven, Dave M, Greg Askew, Zoredache, ceejayoz Sep 22 '14 at 22:58

This question appears to be off-topic. The users who voted to close gave these specific reasons:

  • "Questions must be relevant to professional system administration. Server Fault is dedicated to professional system and network administrators. End user and enthusiast questions are off-topic (contact your system administrator or hire a professional to help you out). Please see the Help Center for more information." – Sven, Dave M, ceejayoz
  • "Requests for product, service, or learning material recommendations are off-topic because they attract low quality, opinionated and spam answers, and the answers become obsolete quickly. Instead, describe the business problem you are working on, the research you have done, and the steps taken so far to solve it." – Greg Askew, Zoredache
If this question can be reworded to fit the rules in the help center, please edit the question.

1

In terms of disks, search for "iops calculator" in your favorite search engine to determine the configuration with the most IOPS for your risk and requirements. We have some databases on blade servers with a SAN connected with FC, and our CPU idles and RAM is used according to the requirement but disk speed is the real bottleneck. You can clearly see the databases struggling once a VM is moved to slower storage. We implement non-SSD SAS disks in RAID10 and the disks can keep up in our setup.

Some databases like to be big with few indexes while others want to be split into smaller chunks, you will have to do some research on database structures for your application.

  • thank you for giving an answer. Already tried this question on Super User but was downvoted as well. Feel like people don't appreciate the fact that I'm really looking for a theoretically-based answer and that the scope of the question might seem general but that is mainly due to having multiple questions in one story. Your answer is incredibly useful! Thanks a ton for taking the time and effort to help me out! – Jean-Paul Sep 22 '14 at 20:14
  • Do you recommend non-SSD SAS disks? From the IOPS calculators on Google, there doesn't seem to be an improvement between SSD and SAS. – Jean-Paul Sep 22 '14 at 20:31
  • 1
    SSD trumps anything at the moment, but you get various versions. You actually get SSD disks that perform similar to non-SSD disks. You also get the incredible PCI Express SSD disks, they use PCI Express as a bus and not SAS. We use HP P2000 SANs, and the SSD disks that you buy for them are very fast but very expensive, we dont really need the super-speed at the moment (and 12 x cheap 7200RPM SAS disks in RAID10 performs better than 8 x 10k SAS disks in RAID5). So configuration is just as important to increase speed. – Shawn Gradwell Sep 22 '14 at 20:37
  • I'm still contemplating whether to buy a seperate server or work with an in-memory database within a workstation. On the other hand, I don't know how I could get 100GB+ of RAM to support a full in-memory database (is it even possible?). – Jean-Paul Sep 22 '14 at 21:06
  • 1
    Sure you can create a RAM disk, some of the new blades use 1.5TB of RAM these days. – Shawn Gradwell Sep 22 '14 at 21:29
1

Storing your historical data in a time series database will likely outperform a typical SQL RDBMS. A lot depends on how much historical data you intend to work with. If you can read your data from disk once, pull out only the symbols you need, generate bars and keep those bars in the memory or one or several machines you can test faster than if you go to disk repeatedly. You may want to look at storing your historical data on Amazon AWS. You can load a huge dataset into memory across a cluster of machines, do your work, shut it down, and only pay for the resources you used.

You may also want to look at:

kairosdb

SciDB

ExtremeDB

DataStax

InfluxDB

  • Very, very helpful! Thank you! I will definitely take a look at those. Regarding the data, I will need to read the data frequently and then throw it away after analysis. That's why I was thinking that in-memory database solutions might not be very advantageous. What do you think? – Jean-Paul Sep 22 '14 at 20:29
  • If you can find a way to keep it in memory during analysis, even compressed, and not go back to disk you're normally in good shape. – Andrew Sep 22 '14 at 20:41
  • What if my database surpasses 1 TB and I only would buy 64GB of RAM? – Jean-Paul Sep 22 '14 at 21:04
  • Try loading it into a time series database and see how you do. If it's 1TB of ASCII quotes it should occupy much less in memory. – Andrew Sep 22 '14 at 21:06
  • 1
    Depends on the ascii of course. Should be a lot but I don't know how much. – Andrew Sep 23 '14 at 1:27

Not the answer you're looking for? Browse other questions tagged or ask your own question.