3

I want to transfer particular types of files from directory on server. So here is the scene:

The directory structure is

companies/compA/filedir/
companies/compB/filedir/
companies/compC/filedir/

So I want to move *.csv files from filedir directories from each company to other server. I want to use ssh for file transfer. Has anyone used ssh for file transfer in shell script? How do you handle this situation?

Improve this question

4 Answers 4

Reset to default
6

Use scp or rsync.

scp uses ssh to copy files to and from connected computers. Sample syntax of copying from the local computer to a remote host:

scp -r companies/compA/filedir [email protected]:/some/remote/directory

You'll need to use public/private keys in order to facilitate logging in if you haven't done that already for ssh.

rsync can use ssh for it's ability to connect to another server. The sample syntax would be:

rsync -avz -e ssh remoteuser@remotehost:/remote/dir /companies/compA/filedir

It will also need public/private keys enabled.

Improve this answer
1

You would actually want to use scp which uses SSH. If you want to use rsync instead, you would want to use the -e ssh argument.

As for scripting it you would probably create either 1) a for loop that iterates through an array that holds a list of all the locations and runs the corresponding scp/rsync command OR 2) you could use a find statement that goes through and executes each scp/rsync command for you.

The find command example would be something like (untested):

find /dir/companies/*/filedir/ -type f -name \*.csv -exec scp {} [email protected] \;

I suggest that if you are using SSH for any type of transfer you setup some sort of public-key authentication, so that you aren't prompted for the password.

I can go into more detail if need be, just trying to get you started and pointed into the right direction.

Improve this answer
8
  • I was assuming that everything from /dir/companies/* was some sort of "compX" naming convention, so I globbed for "everything" - this may not be correct, but I did assume this.
    – drewrockshard
    Aug 24, 2010 at 15:03
  • @drew - while your approach would technically work, it's far from ideal. Imagine the case where each CompA directory has 5000 files in it. That would mean that for each file, you'd need to go through all the overhead of starting and tearing down the scp connection. This would introduce a ton of extra time into the process. Solutions not involving find are much better in this respect.
    – EEAA
    Aug 24, 2010 at 15:05
  • This is why I suggested the for loop +array. I wouldn't use the find command, but some people prefer it.
    – drewrockshard
    Aug 24, 2010 at 15:07
  • how do you set up public-key authentication and use it in the script?
    – yogsma
    Aug 24, 2010 at 15:12
  • yogsma: serverfault.com/questions/2429/…
    – EEAA
    Aug 24, 2010 at 15:17
1

You wouldn't use the ssh program - you should use scp or sftp. But having said that a better solution would be to run rsync over ssh to avoid copying files you already have received - but this is only an option if rsync is available (or can be installed on the remote system).

In order to avoid having to supply a password you'd need to set up a keypair first, then just:

#!/bin/bash

scp somuser@server:/companies/compA/filedir/*.csv /local/compa/filedir
scp somuser@server:/companies/compB/filedir/*.csv /local/compb/filedir
scp somuser@server:/companies/compC/filedir/*.csv /local/compC/filedir
Improve this answer
0

scp companies/comp?/filedir/*.csv 192.168.1.200:/remotedir being 192.168.1.200 the remote server ip.

If you need user and pass you should check the scp manual (man scp), but I believe you can't specify those in the scp command. Instead you should use keys.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .