All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Transport huge amounts of data (+40GB) between 2 home dsl lines? (seedbox, sftp, rsync)?
Howdy,
My old man is into movie editing, he tapes bands at a few local pubs and they publish them on disc. He edits and produces them with Kdenlive, Kino, Qdvdauthor, 2mandvd and recently openshot. I then create the boxes with either inkscape or imagemagick. Recently he has had to edit HD material (1440x1080, h264) which was about 60 GB of raw material. I sometimes convert it, or do other stuff with the edit material as I have a big-ass mac pro here with vegas and final cut pro. To get the materials to one another we use local nas servers, I've set up a NSLU2 here and there, they rsync all the data at night or when we prefer, but it is kinda slow. (home DSL line all < 512kpbs)
I'm looking for a better way to do this. Situations as above, most of the time +40GB of raw material.
My ideas:
Seedbox (http://biohost.info - http://seed-box.me.uk)?
Strato HiDrive
VPS with big disk space?
Drive back there with external USB disk?
It should preferably not cost a lot (<20/month), and have a lot of bandwith..
Any ideas or suggestions?
Comments
What's the physical proximity between you, e.g., same neighborhood; different countries....
40GB+ at 512kbps, you're looking at over a week to move the data, no matter how you do it. As a wise man once said... Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.
your dsl upload speed is slow , no point to use any internet storage - or maybe i miss your point with VPS
Buy a portable hard drive. Mail it to the source. Mail it to the destination.
40 GB at 512 kbit/sec = 10172.5 minutes = 169 hours = 7.06 days.
I think compression is the key here.
40GB is going to take an age to transport between 2 DSL lines even with compression so my solution (assuming the other house is not in driving distance as that would be what I would do
)
Set up a VPS, install OpenVPN Access Server (comes with 2 free licenses) enable lzo compression and then rsync it to the server, which will compress it further during transit
The other solution which is probably not ideal and will probably cost more than you want to spend is to get a second DSL line and bond both of them.
Or failing all of that, span the data over a few DL DVD's or a BR disk and post it first class it will still probably get there faster than the upload + download
Yes.
Binary data won't compress much...
I think you should run a fiber line from your home to his.
@Damian indeed but a few hours saved would be a god send for that volume of data.
And even can get bigger lol
Both of you buy an RDX drive and just post the media its going to be 100% faster, unless you can afford to wait weeks between moving the data and pretty much loosing your normal access to the net during transfer.
@sleddog Its about a 2 hour drive, but when I go there after or before a nightshift (i'm a nurse) in the hospital (which is about halfway inbetween) it is more convinient. We both don't have access to fiber now or in the near future, not in a city in which it is rolled out. Fiber penetration in the netherlands is only good in the bigger city's, not in the more rural area where we are.
@AnthonySmith I've tried to compress it as tar.bz2 or tar.xz, but the data save is at most about 2 gigs, not really worth the time to compress, md5 and hope it comes over good.
But what about a seedbox? Would that work?
Probably quicker to post it.
2GB will easily fit on a flash drive. Just strap it to a carrier pigeon. It works.
$20/month should easily pay for pidgeon upkeep.
Well no mater what solution you try your going to be limited by the fact that you have to upload it from your home dsl and you cant make that any faster, the only advantage of using torrents/seed box solution is that you can stop and start it when ever you want or set a schedule.
2GB is how much compression saved them on the file size, not how big it is :P
So? File-split across a few flash drives and get a bigger pigeon.
With the size/weight of 32GB SDHC cards you don't need too big of a pigeon ;-)
I think the best solution to this problem, is for your old man to get a computer capable of editting HD video. No matter what you do you're bottlenecked at the source, no way around it.
@Raymii do you have uploaded any of these videos on youtube? link?
Brilliant! Now we just need a Howto for a 32GB pigeon (upgradeable to 64 GB).
@sleddog start here - http://www.faqs.org/rfcs/rfc1149.html
and here - http://www.faqs.org/rfcs/rfc2549.html
Good start, but what really needs addressing is the mechanics, e.g., attachment techniques and poop-proofing.
@sleddog nothing some duct tape can't fix
And to reliably send 64GB of data you can use 3x 32GB pigeon carriers, make it RAID5
I.e. p1 has "A", p2 has "B", p3 has "A XOR B"
We can name it RAIP (Redundant Array of Inexpensive Pigeons)
Duct tape is heavy. There's all kinds of lightweight, waterproof tapes used in the medical field for bandaging, which the OP probably has access to.
I.e. p1 has "A", p2 has "B", p3 has "A XOR B"
I'm not sure. Maybe 2 pigeons each carrying 2 cards would be better. That way the full dataset exists independently in two places at anytime.
Yes, but 2 cards are heavier than one. Otherwise i agree, your suggestion offers better data protection
But probably not more than a flash drive weighed in 1999?
Were there any flash drives in 1999? I don't remember
LOOL! at that RFC docs!
Sorry, I meant 2009. That's the date of the story I referenced earlier.
But it seems no-one else is taking us seriously, so I guess the weight of a 2009 flash drive vs. two modern SD cards is moot.