New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
go with local usb or a second pc in the same network with a large enough disk. having fiber is nice, still I'd doubt you'll see more than 30-50MB/s on your way out, as it is too much dependant on geographical location and congestion on the route.
on the other hand 100MB/s locally should be doable much more likely and save a lot of time then...
using
ddwithskipandseekparameters can deal with the "continue from where I left off" (albeit perhaps in a slightly cumbersome way) ...still
rsyncseems to add just a wee bit of data integrity ensurance to the mix via its inherent check-summery - and it's a bit more configurable in the case where it may be desirable to actually throttle throughput (I've had the - fortunately rare - experience of having an IDS triggered by too many packets per second, and so have found it useful to actually dial the data transfer speed back a bit rather than be null-routed by that braindead trigger-happy good-for-nothing IDS)Buy a usb drive and after done, return it with some restocking fee.
Or I have a google teamdrive and I can invite you so that you can use it temporarily. It is unlimited. Speed is a few hundred mbps in and out.
Exactly - plus your data never leaves your network which is always nice.
I assumed that this was not an option and that's why this whole round trip to remote VPS business. Maybe a friend from whom you can borrow a PC for a few days to stuff it with disks and get the copying done.
True - but not really easily scriptable and requiring a fair bit of manual intervention+attention+brains all of which degrade rapidly over time in these kinds of scenarios
Does Google throttle you if the traffic rate is sustained? Just curious.
I do not know if Google throttle me. The speed is between 100 Mbps and 300 Mbps.
You should be worried more about you ISP throttling speed.
so 6 TB at roughly 30MB/s equals in 55-60 hours. or more. one way. 🙄
PMSL :-D
If you can execute the whole thing in 11-12 days total (backup+restore), pick up a Hetzner auction server with 2x6TB disks (even temporary data needs safety)
They only bill you after the 14 days. So Bill=0 if you're averaging 6days for one direction of transfer.
I doubt 6TB in + 6TB out is a big deal for Hetzner network.
Also, I'm pretty sure you'll end up becoming a client after the experience.
Looks like wishosting already chimed in with my suggestion for N.America.
USB3 is 5Gbit/s theoretical, USB 3.1 gen2 is 10Gbit/s.
Now even if you assume it can actually only manage only half that it's still faster than most peoples internet connections.
spinning rust most likely will limit you to ~150MB/s or lower anyway. still faster and probably more stable than unknown speeds to box X with connection Z ;-)
Wow, I appreciate the responses @dragon2611 @vimalware @AlwaysSkint @Falzo @yongsiklee @nullnothere .
Let me clarify. The 100-200 MB/s is via a DD read test on my raid (from home folder). a test on my root (/) (SSD) was over 400 MB/s.
I am going to be able to do direct sync because I will set up a new raid 10 array, add it to the volume group, then remove the old, migrating the extents.
Getting a new mobo and case to cover the storage, which will also serve to allow expandability in the future.
Basically the same as swapping out a single drive in LVM.
However, I cannot make use of any "friends" PC as I have no connections of that sort, and even if I did, an average computer cannot store 5.4-6 TB of raw binary data (imaging).
I don't see myself using Google Drive as I don't have a program that can stream to their servers. The drive programs would copy a local file as Dropbox does. I have a 1 TB drive and insync, so while useful doesn't fit the use case.
I DO plan on booting to a live USB and mounting the drives while doing a snapshot as I still gotta work
.
The whole remote VPS thing I am asking for is only a backup of the snapshot LVM. Ideally, I will not need to download back or use it. Just me not being stupid.
I am interested to know how I can use rsync instead of dd, but it would be a snapshot LVM and not a filesystem level. For reference, I have XFS on the raid.
Appreciate any further input/advice! Thanks
Also found something strange to ring up my ISP on. my upload is registering as 900+ mbs to my ISP's network and 316.02 elsewhere, but download is roughly 15-20 mbps regardless. So I at least have a very fast upload currently
I guess a location close by still would make a lot of sense to utilize as much as possible of this 900+ mbps.
if you can reach 50-60MB/s to one of the suggested providers you're probably good to go...
but DDing into an image of that size? wouldn't do that, imagine shit hitting the fan and you need to restore from that... by? DDing back? with 15-20mbps?
just rsync the f*ck out out of it, maybe in parallel to speed up things. in the end you'll at least be able to access your files individually if something goes wrong or decide/prioritise on the order you want to restore etc.
I'd still rather buy a 6TB usb for 100 bucks and use that locally for the backup. if you don't need it anymore in two or three weeks put it on ebay for 80 and be done with it ;-)
@Falzo if you can find a deal on a usb 3.1 6 tb drive thats not 400 USD then I may go that route. As is I will be talking to my ISP about my download speed.
Gawd, some peeps have it real good, as I type with my 30Mbps/10Mbps, on a good day, connection. ;-)
USB3 8TB drive, $140
https://www.newegg.com/black-wd-elements-8tb/p/N82E16822234349
USB3 is real world 3.2Gbps or 400MBps, but agree with others you're limited to the 150-175MBps of the internal drive.
obviously not gonna happen ;-)
USB 3.1 won't make sense or help at all... as @rajprakash mentioned the speed of the internal HDD won't even make full use of USB 3.0
that's probably why there aren't any cheap offers for external USB 3.1 drives yet, chips are not yet that cheap to be put into single drive cases for none benefit at all (other than marketing maybe)
however you decide, let us know how it goes and with which provider/solution you go etc.
It looks like my internet is fine. Somehow speedtest.net operates differently on chrome vs firefox and even the cli tool that exists.
However, I am not getting better than 25-30 MB/s and peaking at 40 MB/s to @Francisco network on both LV and NY.
It seems I need to find looking glasses to test providers to get the fastest backup connection. fast.com actually peaked me at 1.2 Gbps for a second, so I do have a big pipe
. If I can get an upload to match as my download and upload are the same, I should be in a good position.
I did an upload test to a DO server I have and got the same as the BuyVM server (straight on the modem).
Tested linode and numbers were poop on upload, yet their looking glass was in the 70+ MB DL.
@Falzo any ideas on getting a list of looking glasses to test with?
I'm generally happy to see that on most providers in the US (though it seems easier to routinely get upward of 40-50 MB/s between many providers in Europe)
was pleasantly surprised to see about 100 MB/s between HostHatch Chicago (Psychz) and ExtraVM Montreal (OVH) just now (and extra surprised since I thought the ExtraVM was capped at 500 mbit/s outbound) - so that's going to raise the bar quite a bit for me now I suppose.
EDIT2:
I'm guessing I may have triggered an avalanche of iperf punters (or worse) - testing again shows less impressive results ... I'm going to come back to it at some random point in the not-too-distant future to reconfirm. (But I seen it! I seen it with my own 3 eyes!)
Who mentioned
100 MB/s between HostHatch Chicago (Psychz) and ExtraVM Montreal (OVH)?I mentioned it just a few minutes ago ...
Not sure I understand if there's something more to your question?
(Such as, why didn't I mention HostHatch earlier? Because they were sold out of storage at the time. They might have more in a few weeks.)
I just tested ramnode and got 112Mb/s on their http://lg.atl.ramnode.com/static/1000MB.test
But they def do not offer what I need it seems
Sorry I thought you were responding/reacting to another comment, lol
@Francisco any input on the difference with networks that you know of. Would love to see at least half the speeds of ramnode on upload
@pcfreak30 keep in mind you're sharing a pipe with other people (unless you want to pay for a dedicated gigabit port) ... so if you're pumping 6 TB over the wire for however long it takes, maybe it's not too unreasonable to expect you'd keep the speed down to a "fair use" dull roar. (I say this with all due respect, love, and understanding.)
I would possibly be willing to pay for the bigger pipe if it was not too much just to be able to transfer faster. O/C, if it goes over 100$ then I might as well get a USB 3.0 drive.
Mr. Murphy is always one step ahead of the best laid plans, it seems ... Though possibly some providers would have the capacity to accommodate the transfer at a goodly speed, definitely best to communicate your intention with them ahead of time (as you are doing now ...)
But yes that local drive does seem to be looking like a good option to carefully consider - as it will probably come in handy at some random point in the future as well.
so my initial assumption in my first comment wasn't so far away... ;-)
as written, I think it's not neccessarily just about the link of the provider itself, but the full route that makes your final speed. a bit congestion on some routers in between might already be enough to kill it - even if the provider has enough room on the (still shared) port.
depending on where you are located (mind sharing?) I'd consider the speeds you see with @Francisco quite nice already and if you want to reach more you'd most likely need something even closer to you. still depending on the peerings of your ISP, it could be you never come real close to that 900mbps at all.
sometimes it helps if you can use more than one connection. but also keep in mind, that you might run into other limiting factors like encryption/compression depending how you build the connection itself (ssh?)
I realize this is a hosting forum, so people are quite hot on traditional hosted solutions like a storage VPS or dedicated server. But Falzo's suggestion is, indeed, exactly the right solution for the task.
There's no reason to bother pushing through terabytes of content on a shared pipe for a temporary backup that can (and should) remain local. For a short-term project like this, a local copy will be faster, more reliable, and generally more trouble-free than any remote solution you can find.
I would strongly urge you to consider Falzo's suggestion, despite the initial cost.
Thanks, I found that 140 deal on Amazon and have bought it. Doesn't seem like I'm going to get a 70-100 MB/s upload to anywhere so I get the point.
Will update if I have other issues or questions or what happens
Glad to hear it.
Did you order with next-day delivery? (Just wondering how many gbit/s would that work out to ... lol)
"Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway."
https://en.wikipedia.org/wiki/Sneakernet