All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
How to reclaim lost space on Debian
Hi.
I bought a storage VPS from a provider here and then installed Debian 11 from official iso image.
Problem is, after installation the disk space shrieked and became smaller than that of provider's template Debian 11 OS.
So I mounted gparted and nothing seems wrong.
I've never experienced this problem from other provider's storage VPS.
Here's a screenshot showing the size difference.
I've done some research on stack overflow stack exchange and alike but found nothing worked for me. resize2fs told me there's nothing to do.
I was wondering if this is related to 1024 vs 1000 but then why is the template Debian 11 works different than official Debian 11.
Is this somehow related to system overhead that someone mentioned in a post I found out earlier but there's solution given.
What's the correct size of /dev/vda1 and how can I reclaim the presumably lost 999-983=16GB (not the reserved space of ext4) space without losing data or doing reinstallation? Since the bandwidth is quite limited and transferring 900+GB data took quite some time.
Thanks in advance.
Comments
run
e2fsck -fvy /dev/vda1from that gparted terminal and while the disk is not mounted. fdisk shows the correct full size, so could be something borked with the filesystem...also got nothing to do with the template vs iso thingy, you got a full 1000 GiB right there.
Recently I had to fit as much data as possible onto a 1TB HDD. For testing, I made a 1000000000 byte file with fallocate, created an FS in it, mounted, and got the following usable space:
If you care about the best possible storage efficiency, reformat your partition with JFS or XFS.
Or if your data is compressible, then into Btrfs and mount with
compress-force=zstd.Aside from the above, you can probably reclaim some space in ext4 by doing:
tune2fs -m0 /dev/vda1. But this will only add you some "Free" space, not raise your "Total" from 983G.still can't figure out what's going wrong.
From template I got 999GB shown in df but it's bloated and that's why I manually installed Debian 11
as @rm_ already pointed out maybe the template is simply using xfs as filesystem instead of ext4. given the numbers he posted this comes very close in terms of difference to what you are looking at...
So ext4 file system ate this many space? Gosh. Never noticed this before.
I didn’t choose xfs cuz reserved space isn’t customizable as ext4.
@rm_ does do you recommend xfs as OS volume’s file system?
edit: I checked NVMe VM from same provider. Template Debian 11 is not xfs, just ordinary ext4.
How loudly did it shriek?
I believe that by default, ext4 reserves about 5% of the disk space for the root user
yeah, but that's not what this is about. from the looks of it he already set the reserved space down to 1% ... it's rather about the metadata the filesystem might consume, see the raw numbers from @rm_ above, there is a ~15GB gap between xfs and ext4.
I rarely use xfs, so wasn't aware it can amount to that much either.
still I do not think it's got anything to do with the templates and stuff. you get a disk image provisioned for your VM and that's it. this does not change or at least usually does not get reprovisioned on a reinstall. however, if you would want to be 110% sure you'd need to reinstall from the template and check for used filesystem and disk size (via fdisk).
on another note, you can consider yourself lucky, as your provider gave you 1000 GiB and not just 1000GB ... the latter would be another ~50GB less ;-)
(thought the past tense was shrinked)
Shrank. stupid auto correction.
The difference is that ext4 pre-allocates all space for metadata, so that if you create tons of small files, it has metadata space to store information about them. Other filesystems allocate metadata "as you go". And if you don't in fact have tons of small files, they will not need nearly as much metadata as ext4 cautiously allocates.
It is possible to tune how much ext4 pre-allocates, see
man mkfs.ext4for-i, -Nand probably also-T. But I never tested that, and it requires reformatting anyway. And if you set these too low, you get the risk of running out of those (no more new files can be created).If there's a template with ext4 that gave more than 983G and closer to 999G, then it probably had these settings tuned for mkfs.
If I make a dd image of the whole 983GB VPS, reformat the filesystem to xfs and then restore from it, will anything work as expected? I've never done such migration before and I'm afraid dd won't work properly.
plus dd image will be as large as the whole 983GB so it's hard to find a 1T storage VPS to fill in
Is there any better way to migrate and restore?
You are operating on the wrong layer.
With
ddyou are creating a 1:1 copy/image of your block device (hard disk).This image contains every bit/block of your disk, including your filesystem and (un)used blocks.
You rather want to save all your files within a
tararchive and restore them from there after re-formatting the disk.However completely backing up and restoring a whole OS with
tarcan be tricky and you might have to rebuild your kernel's initrd with support for your new filesystem.Depending on your skills it could be easier to backup only your user data, re-install the system from scratch using the desired filesystem and afterwards restore your data from the backup.
Thanks for the explanation.
I'm not sure if the providers would allow me to tar nearly one terabyte of files and then send them elsewhere. Heavy IO activity over long period of time.
I'm considering the same though: rsync home, /etc, /usr/local/bin directory elsewhere, reinstall with template Debian 11 and restore (presumably less IO usage), which makes me wonder what the fuss about reinstalling with official iso in the first place
I even found out one 1T storage VPS that only gives me 915GB usable space with official iso and default ext4 setups. while most other providers' are 983GB.
Guess we have to take notice about the actual space in the future.
No, a template is a template, it usually won't let you chose the filesystem.
Instead mount and boot from ISO, connect to VNC console and manually install the OS using the installer which will ask you how to partition and format the disk at some point.
Keep in mind that only the size of the virtual hard disk matters.
The actual usable space depends on various factors like partition size and filesystem for example.
And as already mentioned, it's not commonly defined whether a VPS advertised with 1TB disk space actually refers to 1 TiB (=2^40 bytes) or 10^12 bytes resulting in ~9% difference of the virtual hard disk.
Since with hard disk manufacturers 1K is always 1000 and not 1024, it might be valid to use the same conversion also on virtual hard disk sizes.
Otherwise you would explicitly have to look for GiB and not GB values.
I know. I'm wondering what type of usage the template uses.
I'm going to reorder a monthly storage plan and try all these FS listed here FS and see which one fits me best.
Now I'm pretty sure the "lost space" is due to default mkfs.ext4 usage type option which is standard instead of largefile/largefile4.
For storage VPS, largefile/largefile4 could save quite a lot space. After all most people use storage VPS to save tar of backup files. (single large file)
For 1TB, I expect at least 1000GiB allocated space but this one provider (I'm not going to say whom but it's a top provider here) only allocated around 930 GiB for recent offers, sneaky business move huh.
Thank you for your inspiration. I had never considered FS before your reply.
This is what hard drive manufacturers do, unfortunately, to make the drive sound bigger.
1 decimal (base 10) terabyte is 1000 * 1000 * 1000 * 1000 bytes
1 decimal (base 10) terabyte is 1024 * 1024 * 1024 * 1024 bytes (sometimes written as "TiB")
One decimal TB is equal to ~931 binary (base 2) gigabytes
It doesn't make sense using decimal though, since computers use binary, so everything that actually matters uses binary. Physical storage of data on the HDD itself only cares about binary. I only know of one VPS provider that has started doing this - we're probably thinking of the same provider.
We use btrfs with zstd compression at work on systems with lots of source code, and the compression helps a lot. Developer machines with hundreds of millions of lines of source code have something like 35-40% more disk space available thanks to the compression
Yeah. I've tried many storage VPS providers and there's only one by far is doing this.
They used to provision 1000 GiB but switched to 1000 GB recently. However, their allocated RAM is not in decimal. Might as well save on RAM as well to ensure consistency.
Not that I'm asking for 1024 GiB which very few providers do and I appreciate their integrity but 1000 GiB is the bottom line to me. With default ext4 inode settings that's only 915 GiB usable space.