Hot answers tagged pipe
15
To do that, use one extra file descriptor to switch stderr and stdout:
find /var/log 3>&1 1>&2 2>&3 | tee foo.file
Basically, it works, or at least I think it works, as follows:
The re-directions are evaluated left-to-right.
3>&1 Makes a new file descriptor, 3 a duplicate (copy) of fd 1 (stdout).
1>&2 Make stdout (1) ...
15
Edit:
I ran this with your source file in my environment and have the following results:
[root@xt ~]# time tiff2ps test.tif > test.ps
real 0m0.795s
user 0m0.659s
sys 0m0.135s
[root@xt ~]# time ps2pdf13 -sPAPERSIZE=a4 test.ps > test.pdf
real 0m0.592s
user 0m0.513s
sys 0m0.075s
[root@xt ~]# time tiff2ps test.tif | ps2pdf13 ...
11
You can use xargs with -n1 to run a command once for each piped argument
$some_command | xargs -n 1 touch
In the case of touch however which accepts multiple arguments
touch `$some_command`
will probably work for you.
11
You probably want
sed 's/exp1/exp2/g' foo.txt > foo2.txt
Read more at Sed tutorial, Another tutorial, and A small tutorial at Linux HOWTOs
10
You can use pv to do this e.g.
pv file | processor_application
As pv passes it's stdin directly to it's stdout you don't need to use cat.
Edit
As your program is already running then find the PID of the cat process and then look at the contents of
/proc/<PID>/io
which will tell you how many bytes it has written - wchar.
9
Use
find ... -print0 | xargs -0 ...
e.g.
find /path/to -name "*.html" -print0 | xargs -0 grep -l "rumpus"
from the find man page
-print0
True; print the full file name on the standard output, followed
by a null character (instead of the newline character that
‘-print’ uses). This allows file names that ...
9
You do not need to use xargs, because find can execute commands itself. When doing this, you do do not have to worry about the shell interpreting characters in the name.
find /path/to -name "*.html" -exec grep -l "rumpus" '{}' +
from the find man page
-exec command {} +
This variant of the -exec action runs the specified
command on the selected ...
9
This is unlikely to work how you expect. Zip is not just a compression format, but also a container format. It rolls up the jobs of both tar and gzip.bzip2 into one. Having said that, if your zip has a single file, you can use unzip -p to extract the files to stdout. If you have more than one file, there's no way for you to tell where they start and stop.
...
9
While a zip file is in fact a container format, there's no reason why it can't be read as a stream if the file can fit into memory easily enough. Here's a Python script that takes a zip file as standard input and extracts the contents to the current directory or to a specified directory if specified.
import zipfile
import sys
import StringIO
data = ...
8
Have you tried
$ python example.py | bash
It ought to work, as it's a common enough trick. For example, the monitoring tool munin has a node configurator (munin-node-configure) that tells you what plugins you can run, and then takes a --shell flag that makes it spit out a bunch of ln -s commands to link in the plugins, when piped directly to bash.
There ...
7
Absolutely! Pipe Viewer does exactlty that. Just insert it in your pipeline:
cat myfile | pv | processor_application
You can optimize away the cat in the above example:
pv myfile | processor_application
Which has the advantage of providing an actual progress indicator, since pv can determine the size of the input directly. If you do use pv in the ...
7
I only use for ... do ... done for very simple cases.
For more complicated/dangerous scenarios:
command | sed 's/^/touch /'
This does nothing but prints intended commands. Review the results, then perform the same thing piping to sh -x (the -x flag is for debugging):
command | sed 's/^/touch /' | sh -x
7
You usually are only asked if you want to connect when ssh is performing host key checking. Instead of trying to disable by using expect or a pipe, perhaps you could just disable it in your ssh configuration.
Add StrictHostKeyChecking no to your ~/.ssh/config, and ssh will no longer ask you if you want to connect, it will just connect.
6
Kyle's Unix/Linux command does the job of switching the STDERR with the STDOUT; however the explanation is not quite right. The redirecting operators do not do any copying or duplicating, they just redirect the flow to a different direction.
Rewriting Kyle's command by temporary moving the 3>&1 to the end, would make it easier to understand the concept:
...
6
No there is nothing you can do with DNS. There is probably no way for Google apps mail to directly feed to a pipe.
You could either setup an alias in google apps that forwards to an email address that is hosted on your ticket system server.
You could also setup something like fetchmail to retrieve the email via pop/imap and then send the received messages ...
5
I would guess sed still might create the temp file, but the following might do what you want? (Using strace on this might show you if sed creates a temp file or not).
sed -i '/bar/!d' foo.txt
The exclamation inverts the match, d is for delete, so this removes all lines that don't have bar in them.
5
There are other methods you can use instead of wget and curl:
You can use lynx:
# lynx -source http://www.google.com
w3m:
# w3m -dump_source http://www.google.com
and libwww-perl comes with a handy program called GET (as well has HEAD and POST, which do what you think they do)
# GET http://www.google.com
5
You could use cronolog.
cronolog is a simple filter program that reads log file entries from
standard input and writes each entry to the output file specified by a
filename template and the current date and time. When the expanded
filename changes, the current file is closed and a new one opened.
4
If the process is already running lsof has a size/offset column which may be helpful to you -- find the PID of the cat process you want to inspect and then lsof -o -p [PID].
If the process is not running yet, pv as others suggested is a good option (assuming your system has that utility).
4
When you say "a PHP script" do you mean a PHP script on a webserver elsewhere, or a PHP script run on the command line locally?
I've done sending the mail to a website elsewhere using exim4 and curl, by creating a custom transport like so:
send_to_site:
driver = pipe
command = /usr/bin/curl https://example.com/mail.php --data-urlencode "mail@-"
user ...
4
It's not possible with Info-Zip which is the most common OSS implementation. More importantly though, it's not recommended due to the constructs of ZIP archives.
If a change of format is viable to you then consider using tar(1) instead. It is quite happy with streamed input/output and, in fact, expects it by default.
Additionally you can often tell whether ...
4
You can also use perl one liners if you find you want more regular expression features than sed provides. See this link for a comparison. nik's example would look like:
perl -ple 's/exp1/exp2/g' foo > foo2.txt
3
What you want to do is, make unzip take a ZIPped file on its standard input rather than as an argument. This is usually easily supported by gzip and tar kind of tools with a - argument. But the standard unzip does not do that (though, it does support extraction to a pipe). However, all is not lost...
Look at funzip manual page.
funzip without a file ...
3
Is there a reason you can't just do:
unzip -p sample.odt meta.xml
#modify meta.xml
zip sample.odt -u meta.xml
rm meta.xml
Then you don't have to worry about doing wierd gyrations to pipeline everything back in
In response to you comment, the only two suggestions I can think of is
a) Write the file out to a random temp directory ...
3
This suggests you have other problems with your configuration. There's the hint that TransferLog logs/ssl_access_log -directive is inside a virtual host definition, and the rotated log is not. As the ssl_access_log logs authenticated user and rotated log doesn't, I'd guess that the service rotated log is applied to doesn't actually have any authentication ...
3
We are discussing how to implement exactly this feature on the mailinglist for GNU Parallel right now http://lists.gnu.org/archive/html/parallel/2011-01/msg00001.html
Feel free to join: http://lists.gnu.org/mailman/listinfo/parallel
A prototype is now ready for testing: http://lists.gnu.org/archive/html/parallel/2011-01/msg00015.html
3
You could create /etc/exim4/conf.d/router/350_domain_redirect and use the following contents:
domain_fr_redirect:
driver = redirect
domains = domain.fr
data = username@gmail.com
This will work, but is very specific to your problem. An improvement would be to look up the information from a file.
As for everything being in separate files, you can ...
3
process_a | tee >(process_b) | wc --bytes might work. You can then redirect wc's count to where-ever you need it. If process_b outputs anything to stdout/stderr you will probably need to redirect this off somewhere, if only /dev/null.
For a slightly contrived example:
filestore:~# cat document.odt | tee >(dd of=/dev/null 2>/dev/null) | wc --bytes
...
3
You can try with ClutterSSH :
With ClusterSSH it is possible to make a SSH connection to multiple
servers or machines to perform tasks from one single command window,
without any scripting. The ‘cssh’ command lets you connect to any
server specified as a command line argument, or to groups of servers
(or cluster nodes) defined in a configuration ...
3
I normally go for the second option (pipes all the way) unless one of the intermediate outputs is useful to me for another task. For example, if after running Foo against 50k jobs, you then wanted to run Bar against the same jobs, it would be useful to have /tmp/jobs.csv available.
Using pipes all the way gives the system the ability to forget about data at ...
Only top voted, non community-wiki answers of a minimum length are eligible
