Reputation
521
Top tag
Next privilege 1,000 Rep.
See vote count; VIP usercard
Badges
5 23
Newest
 Custodian
Impact
~32k people reached

  • 0 posts edited
  • 0 helpful flags
  • 77 votes cast
Apr
28
comment Transfer large amount of small files
@ewwhite Ooook, so I did test different scenario, using NC, DD, rSync, SFTP, VMWare, UDT/UDR, I suspect I've got some sort of HW bottleneck somewhere as I'm always using about 5MBytes/s bandwith any technics that I should use. I'll talk to our storage area manager.
Apr
27
comment Transfer large amount of small files
I just want to test the best options with default commands before trying to use a faster and better tool but possibly rejected one by the company policies. I swear that I'll test your tool (as it really seems to be the tool that I need) as soon as all my native options will be out of the performance window.
Apr
27
comment Transfer large amount of small files
Hum... I just think of using a combination of Netcat on UDP mode + DD as a block transfer is always faster than a file transfer. What do you think about that solution?
Apr
27
comment Transfer large amount of small files
@ewwhite, Yep, you're right, It's exactly the same things as the pipe use SSH tunnel and checksum. Just tested it, I've got the exact same 5.5MBytes/s throughput.
Apr
27
comment Transfer large amount of small files
I use the hot clone mode, my VMC is not able to see the pseudo devices, I've found numbers of people facing the same issue with powerpath pseudo-devices which can't be cloned as /dev/emcpowerXY device is not detected by VMC regarding some obscure reason. But thanks a lot for the in deep KB post about VMC anyway.
Apr
27
comment Transfer large amount of small files
thank you very much for your answer, I've edited my question to improve the informations regarding my situation as I didn't provided enough insight earlier. Anyway, long story short, VMWare Converter can't clone/find/see all my attached disks as the previous admin used PowerPath.
Apr
27
revised Transfer large amount of small files
improving informations
Apr
27
comment Transfer large amount of small files
Yeah that what I read on your optimization link, the "magic" is not coming from the UDP stack but rather from the UDT library which is something similar to the diagram optimizer provided by silverpeak on its WAN Optimizer solution. I'll give it a try after the sFTP test.
Apr
27
comment Transfer large amount of small files
Thanks a lot for your deep insight on this topic, but what is the difference between UDR-Rsync and a Netcat/Rsync or Netcat/TAR in UDP transfer? As I try on my relatively small test server, with NC/TAR I don't have encryption or other overhead (Except for archives but its not significant) and achieve the test on a 1TB set of data with around 45 Millions of PDF files, my average bandwith is... well... around 2 or 5 MBytes/s on a 1Gbps link.
Apr
27
comment Transfer large amount of small files
Oups, I forgot to mention that I can't attach the old SAN LUNs to the new VMWare Infrastructure. But yeah, originally I would have done it just like that if I had the link.
Apr
27
comment Transfer large amount of small files
No, indeed it was a Typo mistake, I've an average of 50Mbits/s on a 1Gbits/s link and so about 5MBytes/s transfer rate, which is really slow as I already achieved 720Mbits/s transfer on this link with another machine but with large files of around 300GBytes.
Apr
27
awarded  Custodian
Apr
27
reviewed Approve Transfer large amount of small files
Apr
27
asked Transfer large amount of small files
Apr
19
awarded  Popular Question
Apr
14
comment copy LVM partition to new disk
You should probably rather use the LVM Snapshotting features.
Mar
31
accepted Tomcat 7.0 Clustering FarmWarDeployer Not working
Mar
31
comment Tomcat 7.0 Clustering FarmWarDeployer Not working
Yeah, I noticed that a while ago as I said it on my previous answer below my question. But thanks for your answer which is well explain for any other people which are looking for something similar.
Feb
7
comment Dig Command on specific server gives different results than querying without specifying a NS
Cudo for the whatsmydns URL for tracking propagation!
Feb
7
answered How can I create a small cluster and include non dedicated servers?