Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


storage with fast listing for recursive backups
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

storage with fast listing for recursive backups

donkodonko Member
edited June 2022 in Requests

hello, i need suggestions for which service could be the faster for my use case.

i have right now a vps for backups on servarica but it's on canada and my dedi on hetzner, i need make daily recursive backups and i currently store 600gb (most of them images, php, css, js), i send those backups with a cpanel plugin which uses rsync and it's very slow for speed transfer and also for listing files when restore is needed.

currently i'm thinking to switch to jetbackup and use wasabi s3, this could be the right choice? or there is any other alternative on this price range?

i don't want to store my backups with same provider, so do not suggest me hetzner boxes.

Comments

  • JasonhyperhostJasonhyperhost Member, Patron Provider

    we use Wasabi with jet backups and it works very well for us

    if your looking for cheap storage you have backblaze too

    we use both backblaze for Staff PC Backups & Wasabi for client hosting backups

    Thanked by 1donko
  • TerokNorTerokNor Member
    edited June 2022

    It's not only about the provider but also about using the right tool.

    For our backups, (some large databases and many small files around 1TB in total) at Hetzner we use borgbackup:

    • borgbackup to storagebox and also to a couple of providers with storage VPS in europe (you should use more than one)
    • Daily backups end within minutes (only first one can take some time)
    • You can keep many points in time, since borg does deduplication and it's usually very cheap
    • Regularly performing checksum tests for testing the backup integrity
    • borg listing files is very very fast

    A few providers were explored for this job here https://lowendtalk.com/discussion/174386/a-walk-down-storage-vms I tested some and a few others suggested.

    (Servarica is a great provider but I also had connectivity issues from Hetzner)

    Thanked by 1donko
  • NetDynamics24NetDynamics24 Member, Host Rep

    We can offer you 30% lifetime discount in our USA backups plans:
    https://www.netdynamics24.com/client/order.php?step=1&productGroup=12
    We have many clients using jetbackup to send their backups there and the speed is up to 1 gbps.
    You can contact us for more details. We can also help you setup your backups.

  • risharderisharde Patron Provider, Veteran

    Just curious here, is rsync the problem or something else, seems from the responses, borg and jet are doing something more efficiently (???)

  • Well I don't have experience with jet, but I see two things here.

    Probably connectivity issues between hetzner and servarica due to location (I had same issues)

    Rsync is great for what if does but borg does much more since it's intented usage is specifically for backups (many points in time, encryption, compression, deduplication,checksums etc) and it's a lot faster for regular backups.

    Thanked by 1aozsari
  • ehabehab Member
    edited June 2022

    +1 for NetDynamics24 with above plans ; no complains, fast support.

    Thanked by 1NetDynamics24
  • @risharde said:
    Just curious here, is rsync the problem or something else, seems from the responses, borg and jet are doing something more efficiently (???)

    I get the impression that the main problem is simply something that rsync doesn't do and doesn't try to do: it is just a transfer tool and doesn't offer anything to manage listings for restore & such, so the inefficiency being felt is the unaided poking around a resulting filesystem containing backups (if you have many snapshots of many small files, this can easily be an issue). I presume the other tools mentioned do provide some sort of index to aid with partial restores & such, or their own data structure instead of using the filesystem directly.

    rsync itself can be slow over filesystems (or subsets thereof) with many small files, basically do to the required random small IO access pattern, particularly if you ask it to account for hardlinks (where memory and potentially CPU use balloon a bit, on top of any IO bottleneck), or are using a data store with high IO latency (traditional disks, slow network resource, etc.), but I assume the other tools will be affected similarly by these factors.

    Thanked by 1risharde
  • donkodonko Member
    edited June 2022

    i'm trying jetbackup and wasabi s3 (central eu), been 6h 30m and it haven't finished yet the backup xD is still at 77%

    pd: my server is a AX101 :/

Sign In or Register to comment.