I am curious to find out what techniques people are using to store streams of data. More specifically, streams that are not necessarily audio, video or text in case that makes any difference.
The solution we have in our company has been working for us so far, although with the change in requirements we may have to come up with something different. We have custom streams sent to us from clients. The stream messages are sent to a queuing system and from there dispatched to a sharded MySQL cluster. Once stored the data is rarely access by our customers (they get a live feed of the data too). This has worked fine for the past 10 years or so mainly because the application rarely revisited the data in the stored streams (time ordered data). However, with the new features lurking on the horizon clients might want to revisit and recompute old data so the current storage method may no longer be adequate.
I use 'may' a lot because we haven't done technology evaluation or performance studies yet, ie. I don't have an immediate problem to solve, so it's possible that the current system can handle the load but I am curious as to what technologies/methods others are using in similar scenarios.
Cheers, n
