Specify hardware for minimum transfer rates

Question

I will possibly be working on a system that collects images from a vision system and stores them in a dB along with status information. Because the system is tied to a continuous production process running at high speed there will be a requirement for high data throughput rate. What I wanted to know from SF is how you would go about specifying a system that would meet my requirements.

Physically the system is laid out as:

• Camera sends BMP images and a small file with status information via FTP to a directory on computer (note that the camera can only send BMP. Any image compression will have to be done on the receiving computer)
• Computer scans directory for new images.
• On receipt of image, computer inserts it into dB (or moves image to new directory and inserts a reference into the dB). The status information is also inserted at this time.
• dB is used to feed website that allows browsing of images, displays of statistics of image quality over various time periods etc This website may also be hosted on the same computer.

In terms of data rates, the system will need to receive at least roughly 40-80MB worth of images per second (at around 2 MB per image) sustained .

Possible enhancements include splitting the db/webserver into two systems. Only storing file paths in the dB and getting the computer to do BMP to JPG or PNG compression.

So what would be the basic stats I would need to specify to achieve this?

• Network speeds? Dedicated Ethernet between camera and computer?
• CPU type and speed?
• System bus speeds?
• RAM Speed?
• Disk drive types and speeds?

Thanks for you advice

Edit Corrected sizes to read MB

Edit Forget that I mentioned the word "camera" and replace it with "magical-box-that-drops-2MB-files-into-computer-by-ftp"

Edit Feb 24 Sorry the people who answered and it seems like I have been ignoring you. The project got put on hold for a bit when they realized that not all the components in the system actually had Ethernet (Yeah I should be posting on TDWTF)

First bit of news. When told of the total data requirements the specs were backed off. Now I only have to archive 6 or 7 single line text files every second, and only whenever there is a deemed problem archive the full 2MB images. As the production process running all this is supposed to be producing good product then that should be a rare occurrence. Also they shut the line down if there is a several failures in a row .. so the average data throughput will still be low and I can buffer the inserts on disk until I catch up (if needed)

Now for the horror story. While I really appreciate the advice given on how to build a robust system I found out today that "the" (yes - one and only) computer had been bought for the project (and I had no say in its specs at all). I'm sure its a nice computer but my desk is starting to get head shaped indentations when I wonder how the hell this is going to work with a Dell Optiplex 760.

• E8400 Core 2 Duo CPU
• 2GB of memory
• 160 GB HD
• SQL Server Express (which can only use 1GB of memory, 1 core and 4GB max DB size)

I'll pick the nicest answer and award that as my selection

Actually they are all good answers. Pity I can't split my vote.

Answer 1

I've had to have a bit of a think about this one so sorry for the delay in getting back to you.

If I were building this I'd do the following;

• A dedicated DB server (or cluster) of whatever flavour you're most comfortable with.
• Buy a decent sized dual-controller NetApp box, something like a 3140 with 4 x 10Gbps ports. Setup a 1TB-2TB SAS-based vFiler on it as an FTP aquisition portal, files drop into this share and are also accessable via SMB or NFS. Then setup a second much larger SATA (or SAS depending on budget) vFiler shared via SMB or NFS.
• Buy N servers connected to the NetApp via dual 1Gbps or 10Gbps links, these look at the FTP "Watch Folder", assign themselves in control of managing one file-set via a tiny flag file, copy the images to the larger filer and create a DB entry referencing the files final location, once that's complete clear up the originals including the management flag file. Then look for another job, repeat, rinse.

You're going to have to have reasonably powerful firewalls, routers, load-balancers and switches by the way.

This will allow you to survive a single disk controller failing, lots of disks failing and one or more of the aquisition servers failing too. You can very easily split the front and backend filers onto different machines if load becomes too much, you can snap the whole filer for backup purposes too.

You can then build the backend functionality to query the DB and retreive the images as required without affecting ingress performance.

Let me know if you'd like me go into this in more detail.

Answer 2

You'll need to cluster web servers around a central DB server for this purpose. Also, the biggest (and ugliest) chunk of the work here isn't in storing or serving, it's in acquisition at high speed.

You absolutely must to do a round of performance testing before you comitt on this. This may mean getting demo units from hardware vendors. Call one up and have a chat, the sales folks have the authority to arrange hardware demos that you can use for a week or two. IBM and HP are both pretty good for this.

I'm very concerned about the clunky file transfer method (BMP and FTP) being combined with an expectation of high performance. You're going to run into issues with this, especially in terms of how fast FTP can handle connections. For the extra cash you end up spending on servers, you may be able to stump up for more flexible camera units. How are these cameras able to reach an FTP server in the first place? They're ip-cameras?

Ballpark architecture: 2 x 'comms' server handling file acquisition, input of data to the database, then transfer out to the web server or storage on the SAN. Twin-gigabit nics on each, dual quad cores for processing the images, minimal local storage. 1 x DB server. Specs for this just need to be average, depending on where you're going to do your processing, and how much metadata and statistics you want to store, and how many concurrent user connections you will service. 1 or 2 x Web server. Again, specs depend entirely on the number of users you need to support, whether it's a public or private website, and where you elect to handle your image processing.

What you're trying to achieve sounds like it might be a shop-floor production monitoring system of some kind. If that's the case, this is the kind of job that should be left up to the specialists in that field (SCADA etc). Those systems are wicked expensive, safety critical and soforth, and having a dedicated team that is only loosely associated with IT, is a very good thing (tm)

Answer 3

In response to your clarification regarding the amount of data you are capturing this sort of volume is a serious challenge. You indicate this is related to a manufacturing set up of some sort so I'm assuming that it's going to be a 24x365 requirement. I'm only going to discuss the storage volumes here - others have commented on the processing architecture but I think it is important that you get an idea of the scale of the overall solution.

At 40MBytes/sec you are looking at half a petabyte a year if this is a sustained process and you need to retain all of the files on reasonable storage. If you actually need to do that then whatever storage solution you can provide you with far more instantaneous IO performance than you need. At a minimum if I was buying half a petabyte of storage from a vendor I'd want that to be able to provide some nice fast disk storage options as well as the volume. No matter how you build it half a petabyte of disk based storage will be expensive and will take up a large chunk of Rack space. 6x48 drive Disk Array's with 2TB SATA drives will eat up half a 42U Rack and suck down a couple of kilowatts just to keep spinning.

There's no need to actually waste that much disk space though - you want to abandon the BMP format before you commit these to your medium\long term storage. If you want to keep things lossless PNG should save you between 60% and 90% over BMP, if lossy compression is not so much of an issue then JPEG will definitely save you 80-95% so building a processing stack that converts from BMP before you save it off to permanent storage is vital. Even so you will want to have around 50TB of storage for a year's worth of data which wont be cheap but you can find\build that for the price of an average car rather than the Ferarri type costs that something in the Petabyte scale will set you back. If you are buying that much space then it should also be able to easily provide you with the high IO staging capacity you need for the initial camera offload and for your database.

Your database is also not going to be trivial - if these capture rates are sustained (which I assume they are if this is for a manufacturing process) then you will have almost 2 million records added per day - 3/4 of a billion per annum. If each record is only around 1k then your DB will be 600GB. There are plenty of solutions that can handle that but it is not a small DB by any stretch.

If my assumptions about the requirement that this is a 24x7 process with a steady data rate are off then of course all of this is excessive but even if this only runs 8x5 rather than 24x365 you will need a pretty large and robust (expensive) SAN and it would not make sense to then try to cram all of the support services (Web front end\DB\image procesing) into one box.