Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am using the following command:

\cp -uf /home/ftpuser1/public_html/ftparea/*.jpg /home/ftpuser2/public_html/ftparea/

And I am getting the error:

-bash: /bin/cp: Argument list too long

I have also tried:

ls /home/ftpuser1/public_html/ftparea/*.jpg | xargs -I {} cp -uf {} /home/ftpuser2/public_html/ftparea/

Still got -bash: /bin/ls: Argument list too long

ANy ideas?

share|improve this question
    
I am trying to copy all jpgs from 1 directory to another but only new files and ones that have been updated. –  icelizard Aug 19 '09 at 14:11
    
ls is not designed to do this kind of thing. Use find. –  Dennis Williamson Aug 19 '09 at 14:14
    
The problem isn't with ls, it is with the number of arguments the shell is passing to ls. You'd get the same error with vi or with any non-builtin command. –  chris Aug 19 '09 at 16:03
    
But ls is especially not designed to do this: mywiki.wooledge.org/ParsingLs –  Dennis Williamson Aug 19 '09 at 18:27
    
True, but in this case the error isn't because of a parsing error with ls, it is with passing a billion arguments to a new process that happens to be ls. In addition to being an inappropriate use of ls, it also happens to bump against a resource / design limitation of unix. In this case, the patient has both a stomach ache and a broken leg. –  chris Aug 19 '09 at 21:08
add comment

9 Answers

up vote 15 down vote accepted

*.jpg expands to a list longer than the shell can handle. Try this instead

find  /home/ftpuser/public_html/ftparea/ -name "*.jpg" -exec cp -uf "{}" /your/destination \;
share|improve this answer
    
I used find /home/ftpuser1/public_html/ftparea/ -name "*jpg" -exec cp -uf "{}" /home/ftpuser2/public_html/ftparea/ and got the following error find: missing argument to `-exec' –  icelizard Aug 19 '09 at 14:02
    
You are missing the last argument of cp, the answerer told you right. Double check your implementation. Note that in this answer the dot in "*.jpg" is missing, this could lead to misbehaviors (cp a dir named "myjpg" for example). Note then that may be paranoic but safer to specify closely what you are going to copy using -type file (preventing dirs, symlinks and so on to be affected) –  AlberT Aug 19 '09 at 14:30
    
After closer inspection i missed the “\;” to finish the command that -exec should execute. Silly me! –  icelizard Aug 19 '09 at 14:35
    
@AlberT: thanks for the heads re the missing dot. That was a typo. Answer updated. –  Shawn Chin Aug 19 '09 at 15:34
    
It's not that cp can't handle it. The shell can't. –  d-_-b Jul 29 '10 at 6:05
add comment

There is a maximum limit to how long an argument list can be for system commands -- this limit is distro-specific based on the value of MAX_ARG_PAGES when the kernel is compiled, and cannot be changed without recompiling the kernel.

Due to the way globbing is handled by the shell, this will affect most system commands when you use the same argument ("*.jpg"). Since the glob is processed by the shell first, and then sent to the command, the command:

cp -uf *.jpg /targetdir/

is essentially the same to the shell as if you wrote:

cp -uf 1.jpg 2.jpg ... n-1.jpg n.jpg /targetdir/

If you're dealing with a lot of jpegs, this can become unmanageable very quick. Depending on your naming convention and the number of files you actually have to process, you can run the cp command on a different subset of the directory at a time:

cp -uf /sourcedir/[a-m]*.jpg /targetdir/
cp -uf /sourcedir/[n-z]*.jpg /targetdir/

This could work, but exactly how effective it would be is based on how well you can break your file list up into convenient globbable blocks.

Globbable. I like that word.

Some commands, such as find and xargs, can handle large file lists without making painfully sized argument lists.

find /sourcedir/ -name '*.jpg' -exec cp -uf {} /targetdir/ \;

The -exec argument will run the remainder of the command line once for each file found by find, replacing the {} with each filename found. Since the cp command is only run on one file at a time, the argument list limit is not an issue.

This may be slow due to having to process each file individually. Using xargs could provide a more efficient solution:

find /sourcedir/ -name '*.jpg' -print0 | xargs -0 cp -uf -t /destdir/

xargs can take the full file list provided by find, and break it down into argument lists of manageable sizes and run cp on each of those sublists.

Of course, there's also the possibility of just recompiling your kernel, setting a larger value for MAX_ARG_PAGES. But recompiling a kernel is more work than I'm willing to explain in this answer.

share|improve this answer
    
I have no idea why this was down-voted. It's the only answer that seems to be explaining why this is happening. Maybe because you didn't suggest using xargs as an optimization? –  chris Aug 19 '09 at 16:08
    
added in the xargs solution, but i'm still worried the downvotes are because of something blatantly wrong in my details and noone wants to tell me what it is. :( –  goldPseudo Aug 21 '09 at 0:23
    
xargs seems to be much more efficient, as resulting number of command calls is much smaller. In my case, I see 6-12 times better performance when using args then when using -exec solution with growing number of files is the efficiency growing. –  Jan Vlcinsky Oct 9 '13 at 10:06
add comment

That happens because your wildcard expression (*.jpg) exceeds the command line argument length limit when expanded (probably because you have lots of .jpg files under /home/ftpuser/public_html/ftparea).

There are several ways for circumventing that limitation, like using find or xargs. Have a look at this article for more details on how to do that.

share|improve this answer
    
+1 for the good external resource on subject. –  Török Gábor Oct 20 '09 at 11:15
add comment

There is a maximum number of arguments that can be specified to a program, bash expands *.jpg to a lot of arguments to cp. You can solve it by using find, xargs or rsync etc.

Have a look here about xargs and find

http://stackoverflow.com/questions/143171/how-can-i-use-xargs-to-copy-files-that-have-spaces-and-quotes-in-their-names

share|improve this answer
add comment

As GoldPseudo commented, there is a limit to how many arguments you can pass to a process you're spawning. See his answer for a good description of that parameter.

You can avoid the problem by either not passing the process too many arguments or by reducing the number of arguments you're passing.

A for loop in the shell, find, and ls, grep, and a while loop all do the same thing in this situation --

for file in /path/to/directory/*.jpg ; 
do
  rm "$file"
done

and

find /path/to/directory/ -name '*.jpg' -exec rm  {} \;

and

ls /path/to/directory/ | 
  grep "\.jpg$" | 
  while
    read file
  do
    rm "$file"
  done

all have one program that reads the directory (the shell itself, find, and ls) and a different program that actually takes one argument per execution and iterates through the whole list of commands.

Now, this will be slow because the rm needs to be forked and execed for each file that matches the *.jpg pattern.

This is where xargs comes into play. xargs takes standard input and for every N (for freebsd it is by default 5000) lines, it spawns one program with N arguments. xargs is an optimization of the above loops because you only need to fork 1/N programs to iterate over the whole set of files that read arguments from the command line.

share|improve this answer
add comment

The '*' glob is expanding to too many filenames. Use find /home/ftpuser/public_html -name '*.jpg' instead.

share|improve this answer
    
Find and echo * result in the same output -- the key here is using xargs not just passing all 1 billion command line arguments to the command the shell's trying to fork. –  chris Aug 19 '09 at 16:06
    
echo * will fail if there are too many files, but find will succeed. Also, using find -exec with + is equivalent to using xargs. (Not all find support +, though) –  William Pursell Aug 22 '09 at 13:47
add comment

Using the + option to find -exec will greatly speed up the operation.

find  /home/ftpuser/public_html/ftparea/ -name "*jpg" -exec cp -uf -t /your/destination "{}" +

The + option requires {} to be the last argument so using the -t /your/destination (or --target-directory=/your/destination) option to cp makes it work.

From man find:

-exec command {} +

          This  variant  of the -exec action runs the specified command on  
          the selected files, but the command line is built  by  appending  
          each  selected file name at the end; the total number of invoca‐  
          tions of the command will  be  much  less  than  the  number  of  
          matched  files.   The command line is built in much the same way  
          that xargs builds its command lines.  Only one instance of  ‘{}’  
          is  allowed  within the command.  The command is executed in the  
          starting directory.

Edit: rearranged arguments to cp

share|improve this answer
    
I am getting find: missing argument to `-exec' /home/ftpuser1/public_html/ftparea/ -name '*jpg' -exec cp -uf "{}" /home/ftpuser2/public_html/ftparea/ + –  icelizard Aug 19 '09 at 14:16
    
I rearranged the arguments to cp to fix that error. –  Dennis Williamson Aug 19 '09 at 14:51
add comment

It sounds like you have too many *.jpg files in that directory to put them all on the command line at once. You could try:

find /home/ftpuser/public_html/ftparea1 -name '*.jpg' | xargs -I {} cp -uf {} /home/ftpuser/public_html/ftparea2/

You may need to check man xargs for your implementation to see whether the -I switch is correct for your system.

Actually, are you really intending to copy those files to the same location where they already are?

share|improve this answer
    
apologies these are two different directories should be ftpuser1 and ftpuser2 –  icelizard Aug 19 '09 at 14:04
    
Just tried this: ls /home/ftpuser1/public_html/ftparea/*.jpg | xargs -I {} cp -uf {} /home/ftpuser2/public_html/ftparea/ Still got -bash: /bin/ls: Argument list too long –  icelizard Aug 19 '09 at 14:07
    
Oh, you're quite right, of course ls will have the same problem! I've changed to find which won't. –  Greg Hewgill Aug 19 '09 at 20:57
add comment

Go to folder

cd /home/ftpuser1/public_html/

and execute the following:

cp -R ftparea/ /home/ftpuser2/public_html/

In this way if the folder 'ftparea' has subfolders, this might be a negative effect if you want only the '*.jpg' files from it, but if there aren't any subfolders, this approach will be definitely much more faster than using find and xargs

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.