Hetzner Storage Box showing wrong total size (solved)
Dear all,
I run Proxmox on a Hetzner server and did rent one of their "Storage Boxes" to mount it as Samba share and to create the daily Proxmox VM snapshots there.
So I mounted the Storage Box to a directory called "/mnt/storagebox" using:
mount.cifs -o user=XXXXXX,pass=XXXXXXX //XXXXXX.your-storagebox.de/backup /mnt/storagebox
Worked fine. Also added the fstab entry later.
I then created a new storage directory in Proxmox to let it create its VM snapshots to the mounted space. Also worked fine for the first and second day. Because of the limited available space, I set a maximum retention of 2 snapshots for each VM. Also worked fine. Older backups get deleted automatically.
BUT: It seems as if Proxmox cannot see that the space has been free'd. It keeps thinking that the storage space of the deleted snapshots is still occupied and refuses to store further backups because of "missing space". Which is just not true. There is more than 1 TB of free space on the Storage Box after deleting the old snapshots.
Has anyone ever come across a situation like this? Is that a "Samba thing"? I never worked with Samba shares before, therefore I have no idea. I thought that the solution would be perfect for my needs, but it sucks for the moment.
I am grateful for any advice! Thanks a lot in advance!
Kind regards
Amitz
Comments
Just a wild guess, into the blue:
I had problems with samba sometimes if the "recycle bin" feature is activated, then the files are not really deleted but rather copied to some hidden directory.
Well, but if the panel tells you that there is more than 1TB of space left after deleting a file, this might be not the case...
I think I have to contact Hetzner about this.
@Bochi: I have no access to the Samba configuration (all on Hetzner's side) so I cannot de-/activate the recycle bin feature, right?
Hm, that's right of course! Even if it sounds like exactly this "problem" I don't believe it should be configured that way on Hetzner's side.
I assume you already tried remounting and checking the contents from maybe another machine...
So shoot them a mail, they are usually pretty fast in terms of reaction time and problem solutions.
So shoot them a mail, they are usually pretty fast in terms of reaction time and problem solutions.
Yes, I did try this. Unfortunately with the same results. Ticket has been opened, but I will probably not hear back before tomorrow morning. It's Sunday evening here in Germany, and Storage Boxes are not supported over the weekend. Let's see what they say...
It even just shows 400 GB as total space (instead of 2 TB) when I connect as above. Something is wrong.
I agree this seems weird. But I haven't used their storage boxes with samba other then small testing. If I remember correctly you can mount that space via sshfs too, so maybe give that a try and see if there are lost'n'found things or hiden files somewhere?
Can't recall if there is a way to reinitialize the whole box to start from scratch though.
I just mounted the share via sshfs and it also shows the wrong size there and no other (hidden) files that could give me a clue.
I haven't seen any too. The strange thing is: It all worked fine at the beginning. I was shown the full 2 TB of total space. It happened after Proxmox deleted older backups during the process based on my retention settings. Really strange. I am about to order another storage box just to see whether Proxmox messes it up again or if there is another source of the problem.
sshfs is broken, as Hetzner told me 1-2 weeks ago, no idea if patched. Ends in I/O errors....
Holy Shit!
I just found the reason!
Storage Box snapshots (the ones you do in their robot panel) count towards your general quota. I was under the impression that snapshots are indepent from that. I just deleted the 2 existing snapshots and "Bang!" - I had my space back. Seems as if that feature is not too useful then...
Thank you all for the help!
Argh, so "captain obvious" striked again!
Indeed. I really did not expect the snapshot size to be part of the quota. Not that I really need that feature, but it would be way neater if snapshots were an independent thing. You just get a limited amount of snapshots anyway...
However, all good in the end. :-)
So, you are the first one here trying this feature.
Then, the trick of the multiple snapshots to get more and more space doesn't work =P
Yes, I can confirm that now. ;-)
That's very odd and a drawback indeed.
How could a snapshot feature be called that, if you can't really make use of it. How is one supposed to do even a single snapshot of a boxed filled more then 50 percent then?
Sorry for giving a wrong advice on that, I totally fell for their misleading advertising after all
Glad you found the reason anyway...
;-)
Hetzner answered my ticket and is confirming the situation. Here is the ticket reply (in german):
>
>
>
which means that snapshot isn't duplicating content at all, probably only hard-linking stuff if it exists or preserving if it gets deleted in first place.
(I am going to correct my older postings regarding this nownvm, to old to edit ;-))Probably just using the snapshot feature of the underlaying Filesystem they use so the size of the snapshot will essentially depend on how much data has changed between the current state and the state when the snapshot was taken.
I did look at the storage boxes but presumably CIFS is open to the entire internet which makes me feel a little nervous or can you lock it down to certain IP's only
kind of nervous in the way someone could try to connect to your box?
that probably is something hetzner has to deal with on their end (and hopefully already took care of).
as far as I remember you can't set the password yourself only generate new ones which seem to be long'n'strong anyways. also the username won't be guessable easily, so I suppose brute force attacks will be very easy to identify and to be blocked.
they are doing this for a long time. even before the storage boxes, this was the way they offered backup-space for their dedicated servers, so I guess they have some kind of experience on this ;-)
if you are just concerned of possible lack of encryption while using samba itself, you could alternatively mount the storage via sshfs instead.