Skip to main content

Elephant in the Cloud

Today, I came across an interesting issue with cloud. Its called LFN aka Elephant. LFN refers to a TCP/IP issue called Long Fat Network. You can google for it and get the interesting technical details on this subject. What this means for a cloud user is simply that he or she is going to have to wait for ages before his or her fat file gets uploaded to the cloud server. It does not really matter if you have 2 Mbps or 20 Mbps. The latency in the network prevents the TCP from being able to transfer the files at the maximum available bandwidth. 

 And this is serious issue for all cloud users! I tested it across many cloud service providers -  Amazon, Rackspace and Opsource. Over my 4 Mbps, I could not get more than  100 kbps while transferring over scp. 

 This file has the relevant test info for copying speeds on various cloud operators.

 So, whats the Solution?

 Solution#1: Simple and Quick

  • Split the files into smaller chunks. I used hjsplit - it has software for windows as well as Linux. eg. hjsplit filename
  • Use a multi-threaded ftp client. Cuteftp supports muti-threading and has a 30 day evaluation version. GoFTP is another choice. Filezilla is my choice anyday :). 
  • Using such a client, you basically transfer multiple files in parallel. And hence increasing the overall throughput.
  • On the server side, you run hjsplit -j to join the files back after all of the chunks have been uploaded. eg. hjsplit -j filename.001

Solution#2: Technically engaging, Robust and long term - more appropriate for corporate environments with continuing needs for heavy file transfer

  • change TCP/IP settings to allow higher packet size on the file servers. 
  • Use UDT protocol based ftp server and client.

How to achieve the above two solutions is a post for another time.
Labels:

Comments

Post a Comment

Popular posts from this blog

Connecting Ubuntu to Belkin N+ router USB drive

I recently purchased Belkin's N+ Wireless router. It worked great. Easy to set up and comes up LEDs that should help troubleshoot where the problem is. One of the other features is that it comes with a mount point for USB drive. This is good since it just provides an easy way to create a local NAS. You keep the drive in one place and access it from any of your laptops. Connecting it from Windows is easy. The usual \\belkin N+\Drive Name works. However, on linux, it did not work out of box. Looked like trouble with space in the name at first. But when I tried with IP address and it did work. I inferred space may not be the issue. I tried few suggestions from ... http://alittlecharm.blogspot.com/2009/06/connecting-ubuntu-to-belkin-n-usb-drive.html .. this did not work for me. Not as it is, at least. One reason i avoided to implement these suggestion as is that it required to install smb4k which would have also installed base kde and I wasn't in big favor of installing the whole t...
12 comments
Read more

CoLinux - Setting it up on Windows XP

About this article I wanted to write an article thats quick and simple get colinux working on a Windows Host. I have skipped details that can be referred elsewhere in the spirit of keeping this article simple. The infrastructure used: Laptop: Toshiba Portege A 200 Host OS: Windows XP Guest OS: Ubuntu 8.04 (alpha 6) What is Colinux Colinux or Cooperative linux is an easy way to run Windows and Linux simultaneously on the same machine without the hassle of partitioning. More at ... http://colinux.org/ Preparing Setup Space Requirements Ensure you have ample space on your drive depending on how full you want to enjoy linux. Below numbers may help you decide if you have enough space to set up colinux: Bare minimum non-gui colinux: 2G Gnome/Kde based colinux: 5G For use as full time OS: 10G+ Software needed Qemu QEMU Accelerator Module Colinux Wincap Ubuntu or any linux distribution of your choice. I used ubuntu for this article. And you should download the .iso file f...
2 comments
Read more

VirtualBox to KVM Migration

What: Virtualbox is a nice desktop virtualization software from erstwhile Sun Inc.  I used the personal use as well as open edition of it for quite sometime. However, could not resist the temptation to migrate to KVM, given its increasing roars in the virtualization space - both for desktop as well as server side. Not having a good tool to manage the VMs was a shortcoming of KVM for sometime. However, with Virt-Manger, this seems to have changed quite a lot. And so I started to migrate my windows image to KVM based. Source: Host OS: Ubuntu 10.04 Virtualization tool: VirtualBox Guest OS: Windows XP Sp3 Target:  Host OS: Ubuntu 10.04 Virtualization tool: KVM Guest OS: Windows XP Sp3 objective is migrate this guest image to KVM so that the migration is seemless. How: Pre-planning: Shut off windows XP gracefully. It probably might make a difference. But in anycase, will be a good idea rather than land up in surprise You need reasonable space. nearly 3 times of ...
1 comment
Read more