New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
Their is a speed limitation during the route from your peer to another peer.
Try Cloudflare Wrap.
If you run rsync over SSH, you can also try this option:
-e "ssh -T -o Compression=no -x"Beside bbr, for such a long distance you also need to enlarge TCP buffers to obtain good speed. More details: https://lowendtalk.com/discussion/comment/4393248#Comment_4393248
on both servers use newer rsync versions that support zstd compression and xxhash hashing algorithm https://blog.centminmod.com/2021/01/30/2214/fast-tar-and-rsync-transfer-speed-for-linux-backups-using-zstd-compression/
surely putting 100TB through cloudflare warp will have no unforeseen consequences for OP
i saw one mjj provider using cloudflare warp to tunnel their outgoing network lol
I achieve a faster speed using syncthing, worth a try
Does both of the servers support IPv6 with higher throughputs?
Distance is a major contributing factor: see Bandwidth-delay product
You should try some of the suggestions mentioned (not use SSH, maybe disable compression), but if your data is a bunch of small files, you're going to have massive overhead that will exacerbate BDP issues. If you've tons of small files, can you pre-bundle them into tars or zips? Do you actually need rsync, you can stream tar file over SSH, it may be faster.
Lastly, throw parallelism at the problem: tried fpsync? It's a parallel rsync frontend, included in Debian/Ubuntu
sudo apt install fpart. It's what I use for transferring between California and Hetzner DE, I can max out 1Gbit with it.ssh has tiny ass window, over high distance itll limit throughput
use rclone over webdav with hella concurrent transfers if you wanna max out the pipe
they are too far apart. aint gonna get better
webdav in nextcloud?
If you have fast drives (i.e nvme) and enough room, just compress to large tar.gz file, split it into part$ if it too big (1-10 gb per part) and then send it with rsync to another server.
I did the calculation for 500gb of files consisting of images of 500kb-1mb each, rsync between 2 servers is about a day to complete
In comparison, archiving those folder only take 1.5 hours (after splitting) and 30 minutes to transfer.
Maybe get a unmetered server with a 1gig port, relay over that.
Amsterdam or New York, for example.
You literally trying to beam 100TB over the entire planet.
Getting 100MB/sec stable would be nuts.
I tried that with wg-mesh, I added iperf support.
So It would iperf every link and rate them based on the highest speed and route my video streaming over that.
Wasn't as fast as I wanted it though, maybe due to the storage server was already congested.
The eternal questions of compression and hashing algos/programs... *g
I'm confident though that neither plays a significant role in this situation. The problem here IMO is twofold, (a) connectivity (on VPS), and (b) too many software layers involved. If one needs speed the preference should always be "close to the iron, or in this case, close to the kernel" and certainly stay away from the OpenSSL and kids tar pit.
Hence: Do not compress on the fly but locally before sending and use the fastest algorithms/compressors available (zstd is really good and fast stuff but not the fastest in some situations). And first think about and make a decision whether fastest-possible or smallest-possible is the right choice. As your experience suggests, smallest-possible highly likely is the right decision (because the bottleneck is the network transport).
And, if any possible, do not use encryption on the fly (e.g. ssh/scp) but encrypt in advance.
Here is how I would go at it: create reasonable sized tar balls, i.e. do the whole thing *in chunks, which also will be handy in case of occasional failures. Then compress the tar balls with a good compressor (I personally favour lzip4 for such jobs, but zstd with a high compression flag will do the trick too). And finally I'd transfer all the chunks (maybe about 10 GB in size each) to the other server (and then do everything in reverse there).
And yes, try to tune the network parameters on both side as well, e.g. find the best congestion algo for your situation.
And in case you know how to do it, try to script the whole job e.g. in Python or Lua or somesuch.
"rclone serve webdav" on the source server
"rclone copy" on the destination server
if you want a quick dirty solution, nginx to serve the files (just firewall it to your destination IP)
then "aria2c -x 16" everything
when we replicated a few PB across the planet we managed a consistent 40-50Gbit doing this
aria2 is awesome.
you coud setup rsyncd to avoid ssh overhead, and use multiple parallel rsync connection to utilize the full speed.