I’ve got a challenge that as it stands right now, is hitting limitations of cloud computing in terms of IOPS and CPU. The thought is to bring these systems in-house long term but I think it can be architected in a better way to make better use of the resources available.
App A writes anywhere from 100-200+ files per second to the file system. This file system used to be a remote mounted file system but is now being written locally to get the most IOPS we can. We are presently writing to block storage at about 200-300MB/s.
App B remotely mounts this file system and parses these files and pushes data into a MySQL DB. After it performs this function, it deletes the file. This app is extremely CPU intensive. We are working on a rewrite in a more efficient, multithreaded language.
We are working on making the parsers more efficient, but in the interim we need to find a way to improve the whole write/read process.
If I have more than 10 parse servers working on the files, it causes enough IO wait on App A’s server that it tips it over. If we have a central file server, it can’t handle the IOPS either causing extremely high load averages.
Is there better options than writing/reading from a file system?
I’m limited to cloud based product offerings right now and scaling out our present solution to where we need to be will cost us over $1m/yr.