Following the epic fiascos of the two main Bitcoin exchange sites (Mt.Gox, which went offline due to sheer traffic load during the worst market crash in the history of Bitcoins, leaving people with falling prices and a complete inability to sell their assets; and Bitcoin-24, which was apparently ran by someone who didn't even know what a database transaction is), something is becoming increasingly clear: these sites started as hobby projects, when Bitcoin was just some obscure nerd stuff, and were totally unprepared to manage Real Money. Thus, of course, they failed miserably at it.
The reasons are always the same you find in hobby projects who unexpectedly have to scale up to real world needs: lack of design, lack of planning and testing, lack of real programming expertise, lack of knowledge about how to design and operate production mission-critical systems, and so on, and so forth.
My question is simple: how should one properly design such a system? Main requirements would be:
- Security. It should not be subject to any tampering, and people should be able to put their money into it and get it back (unless they lose it while trading, of course).
- Scalability. It should be able to handle the current and projected traffic load for the foreseeable future, and should ideally be able to scale by adding more servers to it as needed, without the need for a complete redesign.
- Speed. This kind of market just needs real time processing.
- Availability. It should simply not go down.
These requirements, of course, translate into lots of design considerations, including hardware specs, software tools (platform, database, language...), application architecture, security, clustering/load balancing... I know there are lots of things involved, and I can think about a whole bunch of them, but the question arises exactly from this: I can not think about all of them, having no direct experience in this field.
Disclaimer: I'm not trying to build an online trading system, for Bitcoins or anything else; I'm just curious about how a production-quality online trading system which handles Real Money should be designed and operated.
I'm posting this question to StackOverflow, too, for the more development-specific stuff.