New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
You can take the backup, throw it into a smartermail install, and be on your way. Smartermail has a free tier that allows 1 domain and 10 users.
File Storageitems are stored as the whole file w/ the correct name/path, making it easy enough to cherry pick.To be fair though, Google Takeout probably doesnt have an "Eat in" option where you can import the backup, do they? I'm assuming it's a one way trip. I tihnk protonmail has an 'import' option.
imapsyncwould allow 2 way push/pull on this as you said, but we do have people that want access to download backups.imapsyncalso requires you have every users login/password, which is rarely the case.We've been scolded in a few tickets for not having the functionality yet (though, the complainers couldn't point to anyone else with the flexibility they're wanting either).
Francisco
I had a similar challenge with my pikapods.com service and settled on Restic + user-provided S3 details. It's still in beta, but we already have several hundred users making use of it with relatively limited support requests.
Here some screenshots and how it works from our docs: https://docs.pikapods.com/manage/backup#incremental-backups-to-s3
Restic is for your own DR needs I guess?
Francisco
Maybe provide multiple options to customers instead of one. To limit bandwidth you can have them pick only one option. And frequency limits like once a day.
In time if there's demand, I mostly wanted to get some ideas down. The S3 PUSH setup was the original plan but wanted to get feedback.
Francisco
@Francisco
With all due respect - and I do respect (and like) you - I think a proper analysis is needed. Before continuing I'll admit right away that I do not even really know what that is about, it seems to at least be about email services (and domains as well?).
Whatever it is I'd draw a clear distinction line. One problem is that a decent provider of bloody course does their own backup and that seems to be taken care of well. The other problem is, and that seems to be what this is about, how customers/users can have and get up at their data to keep backups.
But for that quite different parameters are decisive, namely
You do have one major factor on your side though: the whole thing is not a you problem but the customers problem. It's their responsibility to make backups if needed. Your problem only is to provide a (good) way to do that.
All together that leads to
So, my approach would be to provide ftp/sftp (maybe with a nice https GUI) read-only access to some backup server. If i need backups I can do them easily with basic tools and even with mediocre connectivity and it even can be scripted.
Maybe 1 file per week. And please not zipped or gzipped, (a) bandwidth (via e.g. A|VDSL) isn't plenty or cheap, and (b) why waste it? It's 2025 and zstd nowadays comes with the OS (well, on Unix anyway, about Windows I'm not that sure).
well, if someone migrates from Google, from Microsoft, from MXRoute, from protonmail, from Zoho, etc etc, is the next or new provider who usually deals with the migration, and most of the time is some kind of IMAP sync.
i'm not familiar woith mail providers that allow you to download a backup, and most people won't even know how to use it.
But i guess, this is kind of a limit situation, many hosting providers with familiar panels, limit the size of the account for the backup creation, so it may be an option too. any option you decide should work for accounts up-to XXX GB Size, so if a big client with 10TB space needs a backup, wll they should understand creating that backup (and he downloading LOL) would take time.
No, it's for users. I use Borg for the whole server internally.
Well, the service Namecrane provides is SmarterMail, it's kind of like a Nextcloud with filestorage, calendar, meetings and stuff. So it's not just email.
I do agree with a lot of the points you make, but I also see the technical difficulties with a strict "keep it simple" approach.
For example, the "not everyone has a 10Gb/s connection". That is very true, and that is also why something like export to S3 is a good option. You can send your backup directly to AWS or whatever, and they do have a 10Gb/s connection. There are S3 solutions that are as simple as putting a single php file on your webserver, and that will turn it into a S3 storage. (https://github.com/hochenggang/simple-php-s3-server/blob/main/README-en.md)
But yeah, I agree with a lot of what you say. Simple, standardized solutions are always preferable.
Re S3, frankly I think that's not - or at least should not be - necessary for the customers and falls under "his (@Francisco's) problem" because customers should be - and highly likely are - able to rely on Namecrane having reliable (and preferably dual) backup of their data.
So, for the customers the backup problem should largely be reduced to the usual (simple and relatively small) one for which what I propose should be good enough and versatile (e.g. scriptable, cron'able, etc) - modulo customers with huge file storage but for them my approach should also work well, just to another target (some other storage server) which to implement (as an option) should be easy for Francisco. But I guess S3 as an option would be nice.
Anyway, needing a backup tends to mean that a customer is in a tight and/or problematic situation anyway, so IMO KISS is important, even a priority.
And anyway kudos to Francisco for seriously thinking hard (and caring) about the backup question for customers!
Right. These backups are more for the few users that need it for legal/archival reasons, or would just feel better with the lifetime plans if it was available. Our own DR backups handle things quite well and we've helped recover a few users inboxes (staffer nuking emails, etc) already.
I can parade around saying that we're well covered and all that, but LT plans will always have some grinding their teeth a bit. Customers being able to opt into nightly pushes is pretty simple. I'm certain someone on here is going to stand up
rcloneon some cheapo VM they have and make it do S3<>GDrive proxying and pipe crane backups to that.Shit. that's pretty clever.
Anyway, My bet is 90%, maybe even 95%, of our users will be happy with whatever our public backup policy is. So long as its at least daily, I think we're good. Once we settle on an internal setup we can add a note in our TOS about it.
That's why I've preferred the S3 option. It's well supported/documented and there's countless providers offering it. We need 3 - 4 bits of info and they're off to the races.
No problem. We promised it was something we would address. While we got our DR backups going within a few weeks of cranemail going up, i'm running a bit behind on public access
Thanks everyone for the feedback.
Francisco
I agree with this but build it into your platform so that it is 'self service'. You could probably enable a lot of 'methods', monitor usage and then trim it down - if necessary.
@Francisco we already purchased 500gb crate plan in namecrane website for almost 5 months we are using, my query is can i allocate a 100gb to my new client who likes to store files & office documents only does it will be fast for file upload and downloads?
i am currently using crate 500gb plan under singapore location and my client location will be india?
...
The per-domain restic repos idea feels like the most future-proof path, even if it does assume a bit more tech savviness from users. Being able to mount and browse snapshots without pulling full archives is a big win, and you sidestep the nightmare of generating and transferring huge monolithic files
...
I'm a mere nobody around here, but wholeheartedly agree with this
the fun part of restic is it has the “dump” option which exports a tar or zip stream.
Technically that could be streamed to the end user if they want their single chungus archive (be it they do it in with restic itself or we do it through a browser link).
Francisco
Ah yes, the chungus-as-a-service model. Nothing says "cutting-edge backup solution" like handing users a single, majestic blob via HTTP. Restic: for when you want your data back, all at once, like a digital piñata.
i dont know if this this too naive but here my thoughts
One ui, one midleman app, one folder(backups with files starting with usernames
User logins, requests backup, user gets mail notifying him/her or maybe a code, midleman takes username query folder for files then makes secure links, feeds to ui, user has xx hours to download
People: ok lovable, hi Bolt, do this
I cant tell if you are being sarcastic or not, so I just point out that dumping a "chungus" is just a feature in Restic, it's not the way it normally works.
You have rclone... Add Restic to equation :-)
Depends if you want to have pull or push.
Instead of zip - just map zfs to a loop file as a separate pool and send/rcv to it like any other pool.
Afterall the pool still remains functional, data is compressed and a whole volume is mountable for the client anywhere.
Downloading a whole file - have some utility that uses aria2 - fast with minimal overhead.
Downloading increments - with a dedicated cold file/backup pool you can also give access to client to pick up incremental backups never exiting the zfs world.
In general - what goes to zfs - should stay in zfs.
Another idea would be to make this a paid feature.
Charge a rate for every GB storage used and offer redundancy with different locations and let the costumer decide how much retention is needed.
Rebuilds in case of data loss could be for free when triggered over the costumer panel.
If the costumer chooses to export charge some egress fees. If prices are fair I could see this being popular, especially for business costumers 👍🏼
We must keep backups, so all we're trying to do at this point is make is somewhat accessible to the end user without blowing up our overhead (be it CPU, bandwidth, or storage).
Restic does have a
dumpoption where we can send a zip/gz of the contents.It'd be simple enough and not require much more resources than we've already allocated.
Francisco
Can BuyVM continue to operate? Will the renewal of my purchased VPS be affected?
It'll absolutely continue. There'll be nothing stopping you from renewing, short of you getting terminated or... forgetting to pay the invoice on time
Francisco
Extremely large you say? Than I prefer delivery with a truck. Like amazon storage tank.
I think a full delivery truck for 10TB's a bit much
Though, if someone wants to mail a drive and pay remote hands it could probably be arranged.
Francisco
S3 is an amazing way to get this done from my experience.
I have worked on projects (usually either M&A or bankruptcy related or legal holds) where I have had to dump files or backups extremely quickly.
Getting backups up to S3 buckets in a hyperscaler is usually the fastest possible way to arrange both the storage space and somewhere with sufficient bandwidth to get it done. With multipart uploads, it's trivial to at the very least saturate the link you have.
I have offered SFTP in the past for a customer who was leaving my management and had some data stored with me, but usually I would also throw this in an S3 bucket of their choice and instruct their new suppliers on this (which usually works out well for us all).
okey~Thks~~~
I like the S3 option and I think we should probably offer it.
One of our staffers is pretty adamant that we offer the ability to create an archive on request. Basically you login, pick which backup you want a copy of, it kicks off a backend job to generate it and uploads it into an S3 bucket. You then get a signed download link that's valid for a week.
Francisco
Oh that's a neat idea. I can see that being useful for legal holds or similar. A lot of organisations invest pretty heavily in that sort of thing already (can't count how many times I have been able to get work setting up Veeam to back up and provide easy archive ability for exchange mailboxes and OneDrive/user shares).