Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Rsync Speed
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Rsync Speed

ftnftn Member

Hello,

I want to shift 700 GB of data through Rsysnc from Hostdime (Florida) to SingleHop (Chicago) server. Both server x & y having 1Gbps Connectivity, Private Network enabled.

Can someone please tell how much time it may required to transfer 700GB Data @ 1Gbps connectivity?

Comments

  • omg

  • First off can you honestly try to look up the info yourself first .

    Second depends on hard drive, network peering, network saturation, and method of transfer coupled with cpu.

    As such the best way to tell us try it out with some test files.

  • you've more than fulfilled the posting requirement to post offers by now, can we just get it over with please?

    Thanked by 1doughmanes
  • KuJoeKuJoe Member, Host Rep

    It will take a long time. You will not get 1Gbps sustained speed, most of the files will probably be transferred at speeds of KB/s unless you have a lot of big files in which case you're lucky. I had to transfer roughly 11 million small files the other day using rsync over a 1Gbps connection in the same cabinet and it took almost 24 hours, it wasn't nearly 700GB though so it depends on how many files/directories you have (df -i for a better count).

    Thanked by 1ftn
  • AbdussamadAbdussamad Member
    edited February 2016

    @KuJoe is right about small files. You should create a compressed tar ball of everything and transfer that single large file via scp instead. It'll be faster.

    At theoretical maximum speed it should take 1 hour and 40 minutes to move 700GB.

  • simonindiasimonindia Member
    edited February 2016

    these day for large file transfers

    X 10.1.1.1

    Tarball → Encrypted zip → web server allowed for 10.1.1.2 ip(nginx)

    Y10.1.1.2

    Wget → decrypt the zip file → then extract the tarball


    when you rsync lot of small files then there is no way any one telling you how long it will take

    if you make a tar ball then may be (in the case network is good) 4-5 hours

  • Its actually for one of my client. The old server having 700GB of data cPanel.

    cPanel Transfer Tool is faster than any other type of migration, as its can rsyncs simultaneously with multiple threads (4 - 5).

    I also find this tutorial http://g33kinfo.com/info/archives/584

    Which one I should follow. Any other suggestions are welcome. Both of the servers are having cPanel latest Release. Private Network is enabled for both server, so bandwidth is dedicated, no network drops.

  • jvnadrjvnadr Member
    edited February 2016

    @ftn A genuine question: how can you resell vps service or any kind of hosting service, when you clearly do not know some basic stuff about handling servers and managing hosting? Your initial question is a real noobie question and the person that ask it, not only should not be involved with the hosting industry, but not even handle a server for any kind of product environment...

  • jvnadr said: Your initial question is a real noobie question and the person that ask it

    Not sure if posting to meet the threshold to start posting offers. There's been similar style questions from others appearing to about to jump in the max profits hosting business.

  • @jvnadr Im working for a company and there lot of things I have to do. I understand my questions seems Im a noob (: So what, I just need replies (: You are not going to get my Service offer here just like others, but lot of noobs questions (:

  • @ftn - Please, please tell me you're not taking money in exchange for your services? If you are, are your clients aware they're paying you to ask people on public forums for free support on how to do basic tasks?

  • God help the men and women who host with @ftn

  • jvnadrjvnadr Member
    edited February 2016

    ftn said: Im working for a company and there lot of things I have to do. I understand my questions seems Im a noob (: So what, I just need replies (: You are not going to get my Service offer here just like others, but lot of noobs questions (:

    I am not afraid of you posting here offers. In fact, that would be better because readers of the forum can know that you are a noobie and realize you are incapable of handling any hosting service.
    But if you are selling or involved with any kind of hosting services elsewere, people that are clients of yours probably do not know the lack of knowledge you have.
    I really don't mind people here asking noobie questions. In fact, if I know the answer myself, I usually reply with my suggestions. But, in your first post in this thread, you are talking about a real environment and in the next, about something you have to do for a client of yours. That is proving you do sell/maintain a hosting business/environment and this not right...

    /quit

  • @tehdan said:
    ftn - Please, please tell me you're not taking money in exchange for your services? If you are, are your clients aware they're paying you to ask people on public forums for free support on how to do basic tasks?

    Lol, the identity is hide here, no one will know me, who I am or, to whom I work for. Its real help what I could get from here @ LET (: Noob awakens.

  • do a simple rough calculation... it depends on the network connection between those two servers and how you are permitted to max them out and how much IO you can use on each of them (big files vs. small files)...

    let's say you have a good connection and IO, then maybe you are able to achieve something like 30-40 MB/s on an average. this results in around 2 GB per minute, which means it will take somewhat 6 hours for a full transfer.

    as already mentioned, there are a lot factors on this, you may be able to achieve shorter times down to 2 hours, or even longer up to days...

    but as you can do a second rsync in much shorter time after the first transfer, the whole calculation should not be a lot of concern after all.

    I definitely would not go for taring as you need the additional space on one hand and it'll take a lot of time too. and you need to make sure, that you are able to resume broken interrupted downloads and such things. so rsync should be the proper approach...

    Thanked by 1ftn
  • @Falzo Real help mate. Still working in security and hardening on the new server. Will start the process through Transfer Tool after 2-3 hours later :)

  • If those are cPanel servers, you can run a local full backup, of all accounts, basically, running a backup to the same HDD then transfer those compressed files. Or, you can just use the cPanel Transfer Tool which will do it all for you and you can go to sleep as you'll be able to catch the migration anytime you wish in the very same tool.

    Even though, I would advise you to learn simply by googling and using a private server before messing up your clients websites.

    Thanked by 1ftn
Sign In or Register to comment.