0

I am trying to write a script that allows me to check a directory for files that i will sftp to another server.

When the file is created it creates a .ready file that is supposed to signal a file is fully created and ready to sftp. for example a file named 10312014.13.07.40.27711113 would have trailing 10312014.13.07.40.27711113.ready file as well.

i dont know how to properly check for 10312014.13.07.40.27711113.ready and if it's there sftp 10312014.13.07.40.27711113.

below is what I have so far, it's just a shell of an idea, I'm stuck and any help would be greatly appreciated. I am also trying to send multiple files at once, in a batch like process.

Thanks!

#!/bin/bash -x
source /opt/foobar/$1/config/usrconfig.sh

### foobar environment variables ###
export FTPHOST="sftpserv.foobar.com"                                    # server to ftp to
export FTPUSR="foouser"                                              # ftp user id
export FTPPASS="foo123"                                      # ftp user password
export PATTERN=                                      # pattern for file name to ftp
export USING_MARKER=no                                          # using a marker file???
#export TDEMARKER="X"                                    # pattern for marker file if no files to  ftp
export FTPDIR="/home/foo"  # directory to ftp to on ftp server
export DAEMONLOG=/home/foobaruser/TDEdeamon.log
export DAEMONSLEEP=10
export CURRDATE=`date +%d-%m-%Y-%H-%M`
export READDIR=/home/foobaruser/TDE
#export TEMP=/opt/foobar/$ENV/tmp
export TDE_BACKUP=/home/foobaruser/TDE/backup

cd $READDIR
if [ -a  $READDIR ]
   then
     echo " $CURRDATE files found to send in $READDIR!" >>$DAEMONLOG 2>&1
   else
     echo "no files found in $READDIR to send!" >>$DAEMONLOG 2>&1
     exit
fi
sleep 2

touch tde_batch
for i in *.ready
 do {
 FILE="${i%.ready}"
 [ -e "$FILE" ] || continue
  echo " Going to put ${FILE} in $FTPDIR directory" >>$DAEMONLOG 2>&1
  echo "put ${FILE}" >> tde_batch
  }
  echo "quit" >> tde_batch

  sftp -b tde_batch $FTPUSR@$FTPHOST 

  rm tde_batch  

        cp $i $TDE_BACKUP
            cd $TDE_BACKUP
            rm *.ready
done

I ran the script with a -x below is the output.

It looks like it's running fine until it finds the first file, it echo's the file, writes it to the batch file, then echoes the quit to the batch file, I want to traverse, echo every single file into the batch file first, then echo the quit, then let the batch file sftp.

My logic is clearly wrong, some proper directive would be appreciated!

Thanks! I hope this is better information to help with. Also I see the permission denied error, but I assume that batch needs ssh keys shared and not a password, in the production env, the keys are already shared and that wont be an issue

+ FTPPASS=fooy123
+ export PATTERN=
+ PATTERN=
+ export USING_MARKER=no
+ USING_MARKER=no
+ export FTPDIR=/home/foouser
+ FTPDIR=/home/foouser
+ export DAEMONLOG=/home/foobaruser/TDEdeamon.log
+ DAEMONLOG=/home/foobaruser/TDEdeamon.log
+ export DAEMONSLEEP=10
+ DAEMONSLEEP=10
++ date +%d-%m-%Y-%H-%M
+ export CURRDATE=03-11-2014-10-07
+ CURRDATE=03-11-2014-10-07
+ export READDIR=/home/foobaruser/TDE
+ READDIR=/home/foobaruser/TDE
+ export TDE_BACKUP=/home/foobaruser/TDE/backup
+ TDE_BACKUP=/home/foobaruser/TDE/backup
+ cd /home/foobaruser/TDE
+ '[' -a /home/foobaruser/TDE ']'
+ echo ' 03-11-2014-10-07 files found to send in /home/foobaruser/TDE!'
+ sleep 2
+ touch tde_batch
+ for i in '*.ready'
+ FILE=10312014.10.36.51.52667612
+ '[' -e 10312014.10.36.51.52667612 ']'
+ echo ' Going to put 10312014.10.36.51.52667612 in /home/foouser directory'
+ echo 'put 10312014.10.36.51.52667612'
+ echo quit
+ sftp -b tde_batch foouser@foobar01.foobar.com
Permission denied (gssapi-keyex,gssapi-with-mic,publickey,password,keyboard-interactive).
Couldn't read packet: Connection reset by peer
+ cp 10312014.10.36.51.52667612.ready /home/foobaruser/TDE/backup
+ cd /home/foobaruser/TDE/backup
+ rm 10312014.10.36.51.52667612.ready
+ for i in '*.ready'
+ FILE=10312014.11.15.41.64934033
+ '[' -e 10312014.11.15.41.64934033 ']'
+ continue
+ for i in '*.ready'
+ FILE=10312014.11.22.53.52444830
+ '[' -e 10312014.11.22.53.52444830 ']'
+ continue
+ for i in '*.ready'
+ FILE=10312014.12.34.26.49837220
+ '[' -e 10312014.12.34.26.49837220 ']'
+ continue
+ for i in '*.ready'
+ FILE=10312014.12.36.16.5815654
+ '[' -e 10312014.12.36.16.5815654 ']'
+ continue
+ for i in '*.ready'
+ FILE=10312014.12.47.03.24215656
+ '[' -e 10312014.12.47.03.24215656 ']'
+ continue
+ for i in '*.ready'
+ FILE=10312014.12.49.21.40757399
+ '[' -e 10312014.12.49.21.40757399 ']'
+ continue
+ for i in '*.ready'
+ FILE=10312014.13.01.41.84187558
+ '[' -e 10312014.13.01.41.84187558 ']'
+ continue
+ for i in '*.ready'
+ FILE=10312014.13.07.18.23453400
+ '[' -e 10312014.13.07.18.23453400 ']'
+ continue
+ for i in '*.ready'
+ FILE=10312014.13.07.40.27711113
+ '[' -e 10312014.13.07.40.27711113 ']'
+ continue
+ for i in '*.ready'
+ FILE=10312014.13.26.27.08714191
+ '[' -e 10312014.13.26.27.08714191 ']'
+ continue



[foobaruser@foobar02 TDE]$ cat tde_batch
put 10312014.10.36.51.52667612
quit
0

To check for files you can do following.

find $FILE_PATH -type f -name '*.ready' | while read file
do
  real_file="${file%.ready}"
  echo "put $real_file" >> tde_batch
done
echo 'quit' >> tde_batch
echo sftp -b tde_batch "$FTPUSR@$FTPHOST"
rm tde_batch

You can use lftp to connect to sftp on some server; and instead of passwords you can utilize same ssh keys you use for ssh if you want. To automate operations via ftp you can write script and provide it to lftp. For example you can check here

7
  • Thanks for the input, It would seem that the job creates numerous files at once, so I would need to sftp multiple files, is that possible? I was trying to create a batch file, but it's not working.
    – gkelly1117
    Nov 3 '14 at 16:27
  • sftp is like ftp; except it runs through ssh. So yes you can upload/download multiple files at once. Update your question with details of your work and I'll see if I can help you fix that to get it working. Nov 3 '14 at 16:37
  • I will post that I have so far in an answer.
    – gkelly1117
    Nov 3 '14 at 20:52
  • I think I need guidance on how to properly loop the traversing of files being put into the batch file before echoing quit
    – gkelly1117
    Nov 4 '14 at 15:31
  • I updated answer with script that should be looping over files and in the end adding quit as you outlined Nov 4 '14 at 15:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.