Questions tagged [parallel-computing]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
1answer
24 views

How to launch a PBS job with hybrid MPI/Openmp

I would like to understand how a GROMACS job launched on my SGI cluster with PBS/Torque using a hybrid parallelization MPI/OpenMPI, works. The cluster is hyper-threading enabled and each node has 16 ...
2
votes
1answer
32 views

R scripts that only run sequential, any way to parallelize its execution to multi cores?

i have some R scripts, which only can run in sequential way, cannot be broken into chunks or any parallel library for R or any other language cannot be used. Is there any way i can distribute the ...
1
vote
2answers
2k views

Scan the full filesystem in parallel with clamscan

I run a clamav scan weekly on my servers. There is one server with a raid6 cluster of 30TB of disk space where the scan take more than 24h to run. So I wonder how can I run clamscan on the whole ...
1
vote
1answer
70 views

Execute script with different argument on AWS instances

I have a script which takes multiple arguments and i need to run this script on multiple instances in parallel on AWS. For example, for sake of simplicity, if i have three instances in AWS, i would ...
2
votes
1answer
144 views

Multliple core overload above 100% on Centos7 Supermicro Server

I am running Centos 7 (3.10.0-514.26.2.el7.x86_64) on a supermicro H8QG6 board with 4 AMD 6276 cpus (16 cores), for a total of 64 cores. I use it for scientific computing, and usually everything runs ...
1
vote
2answers
156 views

Execute command then parallelize other commands after completion

I'm looking for a single inline command to execute /bin/first and then when it completes execute the following /bin/p1, /bin/p2, /bin/p3 in parallel after.
1
vote
1answer
54 views

Application caps at 20% utilization per core no matter how many threads are launched

I'm trying to use a Dell Poweredge R900 I got secondhand as a compute pool (4x quadcores gives me 16 cores to run simulations with). It's running windows server 2008 R2 enterprise at the moment. I'...
1
vote
3answers
1k views

GNU parallel doesn't fully utilize my CPUs

I'm running a command like this on my 36 core server (EC2 c4.8xlarge/Amazon Linux). find . -type f | parallel -j 36 mycommand The number of files to process is ~1,000,000, and it takes dozens of ...
-2
votes
1answer
1k views

Difference between cloud computing , grid computing and parallel computing [closed]

There are many definitions out there on the internet , but I am asking here , just because someone may answer it briefly from his/her practical experience with these technologies. How does cloud ...
0
votes
1answer
292 views

how to execute bash script with args in parallel on all the remote server if the remote server in the from of pair

Please read carefully problem first and then give solution................... we are having 3 files in directory myscript.sh (the myscript.sh file contain many function and starts/stop to all-pair/...
3
votes
0answers
22 views

Transparently farm processes out to cluster

Situation: At work, we have an in-house tool for data crunching. When a job is triggered, it starts multiple copies of itself in separate processes and communicates with them in order to crunch in ...
1
vote
1answer
176 views

GNU parallel multpile sshlogin nodes behind NAT

Is it possible to have multiple remote nodes behind a NAT with GNU parallel? Suppose some of a GNU parallel cluster exists behind a NAT (which may or may not be accessible only via a single IPv4 ...
2
votes
3answers
901 views

Execute Shell loop in parallel, but only N workers

We have more than 100 git repos, and sometimes I want to grep over all. To update the repos I use this: for repo in *; do (cd $repo; git checkout master; git pull); done This is quite slow. How to ...
2
votes
1answer
146 views

How to improve the efficiency of gnu parallel to read from a compressed stream?

Is another question extended from the previous one [1] I have a compressed file and stream them to feed into a python program, e.g. bzcat data.bz2 | parallel --no-notice -j16 --pipe python parse.py ...