0

I have a process that inserts 10 million records into a mysql db and is running on a linux box with (4 * 2.13GHz processors + 4 gigs of Memory ). Right now it takes ~2 hours to complete and I'm not sure if the process utilizes all the processors. Using top I increased the priority ( reniced to -20) and the running time was down to 20 minutes.

  • I'd like to make the process utilize all the processors and'd like to know what tool I can use to look it up ?

  • Is there any other way I can throw more cpu/memory to the running process and increase execution speed ?

Improve this question
1
  • 3
    Throw us a bone here dude, you've told us very little, what is this process? what's it written in?
    – Chopper3
    Feb 28, 2011 at 10:57

2 Answers 2

Reset to default
2

You know that if this process is single-threaded, it can only run on a single core, right? Unless you're using a program that is (or you've written to be) multi-threaded, you can be sure that it will not use all CPUs. Doing some quick googling, you may also run into deadlocks, depending on your DB structure, if you do succeed in making a multi-threaded insert. Here's some other advice, too : use the bulkloader.

If those don't help you enough, tell us more about the actual problem that you're trying to solve. Do you need to get your insert done in less than 20 minutes? Less than 10? Less than 5? What business problem are you solving, or restraints are you working under?

Improve this answer
5
  • The above example is just a dummy benchmark and I'm basically trying to solve this : I have a job queue where I get thousands of jobs/sec. The worker process is single threaded and it takes around 0.2 - 0.7secs usually. This worker process also updates db records based on the processing and I just wan't to avoid bottleneck here. What do you propose ? I considered creating a multi threaded worker, but then i was told it doesn't use all the cores and I should use multi processing instead. I was stuck at this point that I couldn't exactly use a tool to lookup if process is using all the cores.
    – dineshbhoopathy
    Feb 28, 2011 at 15:33
  • Who told you that a multi-threaded process wouldn't use all cores, and what was the context?
    – mfinni
    Feb 28, 2011 at 15:36
  • To explain further - if you're using a single-threaded program for this, it will use one core. End of concept, from a sysadmin point of view. The only way you can get around this is by having multiple job queues and multiple processes. Dunno if that will work with your MySQL architecture or if something else will bottleneck, like unique sequences, etc. If you're having trouble getting a multi-threaded program that you wrote to work correctly, this belongs on StackOverflow.
    – mfinni
    Feb 28, 2011 at 15:38
  • @mfinni: Thanks for the hint!! I'll check with stackoverflow :)
    – dineshbhoopathy
    Feb 28, 2011 at 15:44
  • OK - don't forget to mark the answer to your question here.
    – mfinni
    Feb 28, 2011 at 16:12
1

I'd like to make the process utilize all the processors and'd like to know what tool I can use to look it up ?

Shard your input set.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .