Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Mirroring/syncing object storage providers
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Mirroring/syncing object storage providers

We're fairly paranoid about our backups, and usually keep most everything in two geographically diverse cloud providers. So far most of these solutions have involved inhouse tools to accomplish this, but we're rolling out WAL-E across our Citus cluster, and I would like to move away from rolling our own solution to this problem. We use OVH for most everything, and are more than happy to use their object storage as well. If we have a failure, it's much more convenient to restore from OVH services. But that leaves OVH as a SPOF. Let's say we wake up and they've gone bankrupt, well I'd like to have a copy on AWS Glacier as part of our DR. This is the setup we have for most other critical data storage, and I'd like to keep it going with WAL-E.

Comments

  • i would continue with OVH, add Hetzner and create your own local backs ups on external disks.

  • @ehab said:
    i would continue with OVH, add Hetzner and create your own local backs ups on external disks.

    What? We're streaming WAL segments to object storage, so that we can do point in time recovery across a cluster. Unless I've missed something in your comment, this isn't as simple as scp'ing a pg_dump to a cheap Hetzner box.

  • i am sorry i missed reading the part WAL-E. what i understood is you wanted to make sure you don't have a SPOF and i mentioned Hetzner.... anyway maybe others can give recommendations.

  • @ehab said:
    i am sorry i missed reading the part WAL-E. what i understood is you wanted to make sure you don't have a SPOF and i mentioned Hetzner.... anyway maybe others can give recommendations.

    You can say "iwaswrongonce"

  • @jetchirag said:

    i was wrong many times and will make mistakes in future.

    Thanked by 1jetchirag
  • Well, I suppose you could run minio or some other S3/Swift provider on a Hetzner dedi, maybe back that with a Hetzner storage box if you need more space.

    Sounds like you may want some tiering -- e.g., an S3 endpoint for streaming from wal-e, which is then periodically chunked for cold storage in something like Glacier or C14.

  • We ended up rolling our own wrapper around wal-e which puts all the data in whichever object stores we have configured. I was hoping to avoid this, but after looking at a few alternatives (incl Minio) it seemed like the path of least resistance.

Sign In or Register to comment.