1

I am making a dataminer which needs to write to around 50 different files every 30 seconds. Each file is around 50kb. This process will run 24/7, 365 days a year. The dataminer is build on Node.js and also has a website (LAMP) running on the same VPS (Debian).

From my understanding this is not very good to be constantly writing to disk all the time.

Do I risk dramatically cutting the life of the disk? Having the whole system quite slow to respond? Or are the 50 files (50kb each) every 30 seconds is nothing to worry about at all?

1 Answer 1

2

Depends on the situation, but server disks are supposed to be made to run 24/7, and usually are RAIDed so when a drive fails they're replaced. If it 's on a VM (VPS I'm assuming is a virtualized host somewhere) with a host, it's not usually your concern with the drives because you're paying for them to take care of the hardware.

As for the slow response/performance of the drive it depends on the underlying hardware and configuration. If it's a hosted VM somewhere, you won't always have consistent performance unless you're the only guest on the machine. Otherwise you don't know if your neighbors are doing something that will eat resources and bog it down on occasion. You'll not know until you actually do it.

Practically speaking as long as your system has a lot of memory to cache operations you should be fine as disk operations would just be queued up and executed as the subsystem can get to them.

If you're a developer and run into problems with this, depending on what exactly you're doing (creating new files? appending? searching?) you can re-architect the solution if necessary. Are you querying the data? Maybe you should be looking at putting this into a database that can be scaled as necessary.

I wouldn't think 50 small files in 30 seconds would be a problem as long as your memory is okay, but you won't know until you push your provider in real-world use.

2
  • I am writing to the new files. Actually performance wise it should be fine. I was concerned about shortening the disk life dramatically. Since my VPS is RAIDed I guess I have nothing to worry about. Feb 12, 2012 at 18:57
  • 1
    Nope, not as long as the sysadmin is paying attention to it and not running RAID 5. Feb 12, 2012 at 18:59

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .