Questions tagged [batch-processing]
The batch-processing tag has no usage guidance.
51
questions
0
votes
1
answer
33
views
Good technology for a large-scale batch operation on many S3 files in AWS Batch with Spot instances
I have an enormous corpus of text data stored in millions of files on S3. It's very common that I want to perform some operation on every one of those files, which uses only that file and creates a ...
- 99
0
votes
0
answers
79
views
How to write a batch job query that has wait times, dependencies and conditional execution?
I'm trying to write a batch job that executes via task scheduler. The objective is to run these exe jobs sequentially. However, some of these jobs have dependencies on others. for eg.
job1.exe (no ...
- 1
0
votes
1
answer
3k
views
Get the previous day date as variable in batch file from powershell command output
The scenario is like this. I need a previous day date (i.e. Today -1) as variable in a batch file. But if previous day is Sunday (i.e. script running on Monday) it should return the Saturday's date (i....
1
vote
1
answer
694
views
Having issues with mkdir, robocopy, to date\time folders in a looping batch file
I have searched everywhere but cannot find a solution.
Im unsure what is going on here but I desperately need some help in regards to this issue,
Im running a batch file (Script for a game Server ...
- 11
0
votes
1
answer
372
views
Several short consecutive SSH connections
Imagine performing a short repetitive process, which ends by doing a short lived SSH connection to some server (e.g. transferring a newly generated file via SFTP). This means that there will be a lot ...
- 103
1
vote
1
answer
3k
views
Windows batch file scripting: how to get directory named with latest date (yyyy-mm-dd format)?
I have a directory with many subdirectories. Each subdirectory is named with the yyyy-mm-dd format. I'm writing a Windows batch file and I need to grab the directory name with the most recent date and ...
- 305
0
votes
1
answer
3k
views
Server load handling : One single batch request vs multiple requests
I was talking to a developer about different apis for create, update and delete and he was saying that it was better to have a single batch request to server to do operations like create, update and ...
1
vote
1
answer
1k
views
How do I prevent additional jobs from a given user from starting?
With the Slurm workload manager how can I prevent more jobs from user bob from starting? Existing jobs should continue to run. The user should be able to submit more jobs but they shouldn't be able to ...
- 111
-1
votes
2
answers
264
views
How to create a batch file which after some time execute a file but if interrupted execute another?
I want to automate my ms office installation 2007 or 2010 running a batch file.
I am thinking having an options menu 1.Office 2007 2.Office2010
When started I want my timer to start..lets say 20secs
...
- 115
0
votes
2
answers
8k
views
Running supervisord process as root
My goal is to run sudo inside one of the programs:
[program:doStaff]
command=sudo python manage.py doStaff
autostart=true
autorestart=true
stderr_logfile=/var/log/doStaff.err.log
stdout_logfile=/var/...
- 1
1
vote
2
answers
333
views
Containers for HPC batch processing
We are facing the problem that a lot of people want to run different scientific software on our high performance computing cluster.
Every user requires a different set of libraries and library ...
- 13
1
vote
3
answers
7k
views
windows start nginx and php-fpm in background in the same bat
I'm trying to create a bat file on windows 10 to start the nginx and php-fpm in background.
At the moment I can do that but the command window (black window) remains open. If I close the window the ...
- 113
-1
votes
1
answer
35
views
Scheduled tasks
Overview:- I need to download a csv file every 10 mins from an intranet site. This is done using a batch file.
To ensure that it happens with out any delay, i have set up multiple tasks using Windows ...
1
vote
2
answers
1k
views
How to tell if a DOS batch script is being run by task scheduler?
How can a DOS Batch command file determine if it is being run by Task Scheduler?
- 51
0
votes
0
answers
61
views
Enterprise batch processing of application-generated emails
Here is what I perceive is our problem.
In our corporate environment, we have developed over 40 custom .Net applications (and growing). A good many of these send email notifications in some fashion. ...
- 101
1
vote
1
answer
2k
views
How to catch "The system cannot find the file" within For /F loop?
How to catch "The system cannot find the file" within For /F loop?
I would like my code to do the following:
1 - Search for the specified file
2 - If file not found, handle the "The system cannot ...
- 11
34
votes
2
answers
4k
views
Why is my bare-metal 16x 2.93GHz cores computer performing poorer than a VPS with 4x 2.5GHz cores?
I have a written a piece of multi-threaded software that does a bunch of simulations a day. This is a very CPU-intensive task, and I have been running this program on cloud services, usually on ...
4
votes
2
answers
9k
views
Version v4.0.30319 of the .NET Framework is not installed and it is required to run version 3 of Windows PowerShell
I have a server which we are using for Batch processing.
I can login to the server with svc_account and run powershell -f file.ps1 and it runs fine in either version 2 or 3 by the following -
...
- 151
44
votes
10
answers
73k
views
Most efficient way to batch delete S3 Files
I'd like to be able to batch delete thousands or tens of thousands of files at a time on S3. Each file would be anywhere from 1MB to 50MB. Naturally, I don't want the user (or my server) to be waiting ...
- 543
1
vote
1
answer
149
views
Will a Scheduled Task On Windows Server 2008 R2 work correctly if windows are opened and closed?
I have a batch script that I am currently running on my own Windows 7 computer as a scheduled task. The task runs nightly when my computer is locked. The task launches an Progress OpenEdge session ...
- 13
-1
votes
1
answer
430
views
Configuring / Securing credentials required by batch job
I have a batch job which should run as cron job on a Linux server. This job requires some credentials to do it's work. Idealy I don't want to have that kind of credentials stored on my server, but ...
- 149
1
vote
1
answer
2k
views
Batch password change of all email in a cPanel account? [closed]
Does anyone know of a way to reset the password of one or more email addresses in a cPanel account from the command line?
- 131
1
vote
0
answers
29
views
My transactional overnight batch has spiked in duration
I have an overnight batch run which is transactional...picking up data doing some small calculations and putting it somewhere else and it was finishing around 3am every day for the last 2 months. Now ...
- 241
2
votes
3
answers
14k
views
Batch reset all iLO passwords via command line
I would like to batch reset all the iLO passwords on our HP Blade chassis without having to login to each one via the HTTPS website. How can I reset the root and administrator iLO passwords via the ...
- 5,620
0
votes
1
answer
8k
views
batch script to copy files as within SMB\CIFS protocol
I'm trying to copy files from remote machine using construction like this
copy smb://DOMAIN;USERLOGIN:[email protected]/folder/ C:/tmp/
it shows next message -
The filename, directory name, or ...
1
vote
2
answers
1k
views
Low priority batch processes on live Mongo database
We want to run MapReduces on our live Mongo database, mainly so that we can extract metrics. However, we've had some bad outages caused by these MRs bogging down the Mongo server (in particular 100% ...
- 205
2
votes
4
answers
8k
views
How can I hide ms-dos window when running a .bat file?
I am running a .bat file for my script (Scheduled Tak (CronJob)) per minute. When it run, seems ms-dos window short period of time. My batch code like this;
@ECHO OFF
C:\wamp\bin\php\php5.4.3\php.exe ...
- 35
-1
votes
1
answer
1k
views
How to reliably run a batch job every 5 seconds?
I'm building an application where the sending of all notifications (email, SMS, fax) will be asynchronous. The application will write the notifications to the database, and a batch job will read these ...
- 4,527
1
vote
3
answers
280
views
Is there a way to see the path of a program executed from the Windows CLI?
If I type a command in the prompt, the command processor searches it in current path, and then (if not found) in the paths listed in PATH environment variable.
So is there a way to easily know in ...
- 111
0
votes
1
answer
334
views
How to delay and queue commands for execution at a later time? [closed]
My site receives numerous uploads and I can't afford to waste cycles and time processing them on the spot (e.g., trimming, optimizing, resizing, etc.). What technologies exist to delay and queue these ...
- 13
-2
votes
1
answer
103
views
Does anybody wish to help a poor researcher publish a paper? [closed]
I don't know if this is a good place to beg for help, but here it goes: basically, I need to run a secure recommender system simulation (C++ console application) in order to meet tonight's deadline, ...
- 222
0
votes
2
answers
1k
views
Moving files with batch files from one pc to a server, to a another pc - worried about disk corruption
I use scheduled tasks that calls a batch file, that calls more batch files to move about three files from a pc, to a server, then to multiple other pcs. It all happens very quickly, as they are small ...
- 43
0
votes
1
answer
474
views
Distribute computing jobs across different machines?
I'm looking for a way to distribute computing jobs (mainly rasterization work) across multiple, idle machines on the network. The servers are running Debian 6 right now.
Most of the work that needs ...
- 131
0
votes
1
answer
47
views
Batch Updates to Numerous Server 2008 Machines?
We run a lot of machines that are hosted in various datacenters (takes out the LAN based options) and we are finding the need to change Administrator account passwords for RDP (we use the same ...
- 650
0
votes
2
answers
4k
views
How to schedule batch file to run on the desktop in win 2008 R2
I have some backup batch file scripts which I was running in xp and I have migrated to windoes server 2008 R2 where it's not running in front when scheduled with task manager. I struggled a lot to ...
- 1,723
41
votes
6
answers
39k
views
prevent xargs from quitting on error
According to the man page, xargs will quit if one of the execution lines exits with an error of 255:
If any invocation of the command exits with a status of 255, xargs will stop immediately without ...
- 2,598
0
votes
1
answer
88
views
Whats the best way to execute high processing tasks on low RAM VPS
I have one VPS on internet and that have only 768MB of RAM.
Now i have run crawlers and various Optimzation algorithms /Data Mining Algos on the one mysql database.
I think i need better computer for ...
- 561
3
votes
2
answers
5k
views
Torque jobs does not enter "E" state (unless "qrun")
Jobs I add to the queue stays there in "Queued" state without attempts to be executed (unless I manually qrun them)
/var/spool/torque/server_logs say just
04/11/2011 12:43:27;0100;PBS_Server;Job;16....
- 841
0
votes
1
answer
455
views
Only half of my pbs/Torque jobs are being scheduled
My supercomputing center recently moved from SGE to pbs/Torque. Now, when I schedule my array jobs, only half of the jobs in the array get scheduled. When they finish, the other half get scheduled. ...
- 2,098
2
votes
2
answers
336
views
Batch rename via command line?
I have a Media Temple (dv) server which I am setting Magento up on.
I have uploaded a few thousand images to the /media/import directory that the client has provided me with. The problem is, some ...
- 2,717
0
votes
2
answers
139
views
Batch File Printing
I have a user that has to print out about 220 documents averaging 2-3 pages each. When printed using the standard Windows method of Select All - Right-click - Print it overwhelms either the print ...
- 111
2
votes
2
answers
1k
views
X11 for apache user
We are using inkscape to convert SVG images uploaded to our server via a web form. For this inkscape offers a batch mode via the -z option, but this batch mode has a flaw: When inkscape is run by the ...
- 185
2
votes
1
answer
2k
views
VirtualBox on Windows Hosts | Export and Import batch automate
VirtualBox on Windows
I am planning to deploy virtual box on around 10 computers (Vista / 7), I have setup a centos vm and exported it as an appliance with the ovf and vmdk files in the same folder.
...
- 2,357
6
votes
9
answers
17k
views
Remove files that are listed in a text file
I have a file that exported a bunch of file names that need to be removed. I need to know how to go about removing each file without having to issue it one at a time at the command line.
I've ...
- 1,763
0
votes
3
answers
4k
views
Zip and FTP a File via a batch script
I am trying to ZIP and FTP a file in one batch file. The Zip part works, but the FTP application reads the entire batch script, rather than just the lines below the FTP commands, causing errors
If ...
- 355
1
vote
1
answer
726
views
How to batch edit a list of files?
I have a list of files where I need to remove some lines that have been added yesterday by a spambot. The section I want to remove looks like this:
^M
<script>[...] bunch of malware code [...]&...
3
votes
1
answer
318
views
Backup multiple Exchange Accounts without direct access to exchange server
For e-mail, we use Microsoft Exchange and it is hosted by 1and1.com.
We have about 30 Exchange accounts that I would like to backup to a PST file. That is, for each account that we have (all 30), I ...
Mike Wallace
0
votes
2
answers
378
views
Windows Job Scheduler
I'm looking for a Windows Server job/task scheduler for a small business. We need more functionality than the built in tools, including SQL Server Agent. We need conditional execution and would like ...
3
votes
7
answers
225
views
Software for Utilizing Idle Workstations on Network
I work at a company that maintains a network of about 200 Windows XP workstations, and several of them are on (locked or logged out) with an idle processor and 2GB of RAM that is just sitting there. ...
- 135
3
votes
1
answer
8k
views
Running 'top' with 'Irix mode' off in batch mode
Can anyone tell me how I can tweak the following top command so that it always runs in Solaris mode in a batch mode operation?
top -d 3 -n 2 -b -p <PID>
I need 'top' to output a process' CPU ...
- 249