0

I need to transfer webserver-log-like-files containing periodically from windows production servers in the US to linux servers here in India. The files are ~4 MB in size each and I get about 1 file per minute. I can take about 5 mins lag between the files getting written in windows and them being available in the linux machines. I am a bit confused between the various options here as I am quite inexperienced in such design:

  1. I am thinking of writing a service in C#.NET which will periodically archive, compress and send them over to the linux machines. These files are pretty compressible. WinRAR can convert 32 MB of these files into a 1.2 MB archive. So that should solve the network transfer speed issue. But then how exactly do I transfer files to linux? I could mount linux drive on windows server using samba, or should I create an ftp server, or send the file serialized as a POST request. Which one would be good? Also, I have to minimize the load on the windows server.
  2. Mount the windows drive on linux instead. I could use the mount command or I could use samba here (What are the pros and cons of these two?). I can then write the compressing and copying part in linux itself.

I don't trust the internet connection to be very stable, so there should be a good retry mechanism and failure protection too. What are the potential gotchas in these situations, and other points that I must be worried about?

Thanks, Hari

Improve this question
1

5 Answers 5

Reset to default
1

You could transfer using sftp (SSH). Simple script can be created via windows to linux - this is probably the easiest way, with some added security.

Improve this answer
1

You can also try WinSCP, this is the easiest way I found to transfer from Linux/ESX machines to Windows, provided they can be connected. WinSCP is free, safe, easy to use, no installation required, very less resources required and real quick in work.

Improve this answer
1
  • I too dont agree for using DropBox. Its easy and tempting to use, however, its neither safe nor reliable.
    – whizkid
    Jan 17, 2011 at 8:48
0

Since you are looking for reconnect and resume, I would use FTP or SFTP, together with one of the many commercial products that can easily be googled. Samba is less of a good solution, since disconnecting might cause problems.

My first find on google was BatchSync, which supports both FTP and FTPS/SFTP, but I don't personally know this product. No doubt that it is not the only one of its kind.

Improve this answer
0

Install Deltacopy on the Windows machine and setup a scheduled rsync job to sync the folder with the logs. Instructions on the site also show you how to run the transfer via ssh - just one note that after setting up Deltacopy you should do a one-off run of the Deltacopy-generated rsync command manually at a command prompt so you can reply to the prompt to save the remote system's key.

Improve this answer
-1

The very easiest way is install dropbox on the windows server and the linux server and just save your log files to your dropbox: http://db.tt/Jw6XNac

Improve this answer
2
  • Transfer production log files through DropBox... Not a good idea
    – sebthebert
    Nov 29, 2010 at 0:33
  • yes you are right 500 million users are all in extreme danger.
    – edelwater
    Sep 29, 2021 at 4:21

You must log in to answer this question.