Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a process that inserts 10 million records into a mysql db and is running on a linux box with (4 * 2.13GHz processors + 4 gigs of Memory ). Right now it takes ~2 hours to complete and I'm not sure if the process utilizes all the processors. Using top I increased the priority ( reniced to -20) and the running time was down to 20 minutes.

  • I'd like to make the process utilize all the processors and'd like to know what tool I can use to look it up ?

  • Is there any other way I can throw more cpu/memory to the running process and increase execution speed ?

share|improve this question
3  
Throw us a bone here dude, you've told us very little, what is this process? what's it written in? –  Chopper3 Feb 28 '11 at 10:57

2 Answers 2

You know that if this process is single-threaded, it can only run on a single core, right? Unless you're using a program that is (or you've written to be) multi-threaded, you can be sure that it will not use all CPUs. Doing some quick googling, you may also run into deadlocks, depending on your DB structure, if you do succeed in making a multi-threaded insert. Here's some other advice, too : use the bulkloader.

If those don't help you enough, tell us more about the actual problem that you're trying to solve. Do you need to get your insert done in less than 20 minutes? Less than 10? Less than 5? What business problem are you solving, or restraints are you working under?

share|improve this answer
    
thanks @mfinni. –  dheena Feb 28 '11 at 15:26
    
The above example is just a dummy benchmark and I'm basically trying to solve this : I have a job queue where I get thousands of jobs/sec. The worker process is single threaded and it takes around 0.2 - 0.7secs usually. This worker process also updates db records based on the processing and I just wan't to avoid bottleneck here. What do you propose ? I considered creating a multi threaded worker, but then i was told it doesn't use all the cores and I should use multi processing instead. I was stuck at this point that I couldn't exactly use a tool to lookup if process is using all the cores. –  dheena Feb 28 '11 at 15:33
    
Who told you that a multi-threaded process wouldn't use all cores, and what was the context? –  mfinni Feb 28 '11 at 15:36
    
To explain further - if you're using a single-threaded program for this, it will use one core. End of concept, from a sysadmin point of view. The only way you can get around this is by having multiple job queues and multiple processes. Dunno if that will work with your MySQL architecture or if something else will bottleneck, like unique sequences, etc. If you're having trouble getting a multi-threaded program that you wrote to work correctly, this belongs on StackOverflow. –  mfinni Feb 28 '11 at 15:38
    
@mfinni: Thanks for the hint!! I'll check with stackoverflow :) –  dheena Feb 28 '11 at 15:44

I'd like to make the process utilize all the processors and'd like to know what tool I can use to look it up ?

Shard your input set.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.