I will possibly be working on a system that collects images from a vision system and stores them in a dB along with status information. Because the system is tied to a continuous production process running at high speed there will be a requirement for high data throughput rate. What I wanted to know from SF is how you would go about specifying a system that would meet my requirements.
Physically the system is laid out as:
- Camera sends BMP images and a small file with status information via FTP to a directory on computer (note that the camera can only send BMP. Any image compression will have to be done on the receiving computer)
- Computer scans directory for new images.
- On receipt of image, computer inserts it into dB (or moves image to new directory and inserts a reference into the dB). The status information is also inserted at this time.
- dB is used to feed website that allows browsing of images, displays of statistics of image quality over various time periods etc This website may also be hosted on the same computer.
In terms of data rates, the system will need to receive at least roughly 40-80MB worth of images per second (at around 2 MB per image) sustained .
Possible enhancements include splitting the db/webserver into two systems. Only storing file paths in the dB and getting the computer to do BMP to JPG or PNG compression.
So what would be the basic stats I would need to specify to achieve this?
- Network speeds? Dedicated Ethernet between camera and computer?
- CPU type and speed?
- System bus speeds?
- RAM Speed?
- Disk drive types and speeds?
Thanks for you advice
Edit Corrected sizes to read MB
Edit Forget that I mentioned the word "camera" and replace it with "magical-box-that-drops-2MB-files-into-computer-by-ftp"
Edit Feb 24 Sorry the people who answered and it seems like I have been ignoring you. The project got put on hold for a bit when they realized that not all the components in the system actually had Ethernet (Yeah I should be posting on TDWTF)
First bit of news. When told of the total data requirements the specs were backed off. Now I only have to archive 6 or 7 single line text files every second, and only whenever there is a deemed problem archive the full 2MB images. As the production process running all this is supposed to be producing good product then that should be a rare occurrence. Also they shut the line down if there is a several failures in a row .. so the average data throughput will still be low and I can buffer the inserts on disk until I catch up (if needed)
Now for the horror story. While I really appreciate the advice given on how to build a robust system I found out today that "the" (yes - one and only) computer had been bought for the project (and I had no say in its specs at all). I'm sure its a nice computer but my desk is starting to get head shaped indentations when I wonder how the hell this is going to work with a Dell Optiplex 760.
- E8400 Core 2 Duo CPU
- 2GB of memory
- 160 GB HD
- SQL Server Express (which can only use 1GB of memory, 1 core and 4GB max DB size)
I'll pick the nicest answer and award that as my selection
Actually they are all good answers. Pity I can't split my vote.