I'm running Snort in conjunction with MySQL for logging, which is generating ENORMOUS datasets (currently the event table is over 2.5 million, I don't know exactly how much because it only goes up to 2.5 million before it clunks out from using too much memory).
Unfortunately, this data is not very useful anymore because I can't pull it out anywhere else (the stored procedure causes the server to crash).
My question is, is there a way to optimize MySQL for these huge datasets or is this beyond the technical capabilities of MySQL and I need to go to something like Oracle, MS SQL, or PostgreSQL?
We have both an Oracle and a MS SQL Server instance but both of these are business critical production servers and it would be very bad news to knock either one of those offline or inhibit their abilities.
Any thoughts on the matter?