All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
[Tutorial] Turn a cheap HDD storage VPS into NVMe speed
Yep, you read that right.
This method will make your storage VPS feel like a local NVMe drive, assuming your local disk is using NVMe. Works on VPS and linux home desktop.
Real world result:
- Before (without cache)
dd if=/dev/zero of=./testfile bs=1M count=100
104857600 bytes (105 MB, 100 MiB) copied, 29.1344 s, 3.6 MB/s
- After (with cache)
dd if=/dev/zero of=./testfile bs=1M count=100
104857600 bytes (105 MB, 100 MiB) copied, 0.144608 s, 725 MB/s
The system used
rclone
Rclone is opensource, compatible with mounting many big names: Google Drive, Dropbox, Azure, Amazon S3, etc. In this tutorial, we will mount it using SFTP.@Clouvider Debian 12 as VPS server.
I'm using Debian and logged in as root. If you're not logged in as root, addsudobefore each operation.@labze (HostBrr) StorageBox as Storage VPS.
This method also works with any other providers that support SFTP, you just need to change the port to 22.
Before continue, make sure your server hardisk has at least 10GB available.
1. Installation
apt install rclone fuse
Create dirs.
mkdir /mnt/storage
mkdir /mnt/.cache
2. Configuration
If you're running it on a VPS, you can use:
rclone config
If you're running it on home desktop, you can open a web GUI, using:
rclone rcd --rc-web-gui
Then fill this input:
To make it easy I will use the email content I get from HostBrr as reference.
name = storage-sftp
storage type = sftp
host = [Storage Server IP-address]
username = [Username]
port = [Find this: You must use port ***** for SSH]
password = [Password]
Ignore the rest of inputs.
Fill the port with 22 if you're not using hostbrr.
3. Running the rclone
rclone mount storage-sftp:/home /mnt/storage --daemon \
--dir-cache-time 72h --cache-dir=/mnt/.cache \
--vfs-cache-mode full --vfs-cache-max-size 10G
And boom! your storagebox is now available locally!
You can now browse and copy paste on /mnt/storage just like it's a local storage. You can also watch videos from this storage with minimal hiccups.
No need to wait the upload to finish, you can edit the files on the fly while they're being uploaded in the background.
But it's not done yet, because when you reboot, rclone will close and you need to run it again.
4. Run rclone at every boot
These steps will make sure that the rclone service runs after booting, using systemd.
First, make sure the directory is unmounted.
fusermount -u /mnt/storage
Create systemd service.
nano /etc/systemd/system/storage.service
Paste this, edit if needed:
[Unit]
Description=rclone mount
AssertPathIsDirectory=/mnt/storage
After=network-online.target
[Service]
Type=notify
ExecStart=rclone mount storage-sftp:/home/ /mnt/storage \
--dir-cache-time 72h --cache-dir=/mnt/.cache \
--vfs-cache-mode full --vfs-cache-max-size 10G
ExecStop=/bin/fusermount -u /mnt/storage
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
Then enable and restart the service:
systemctl daemon-reload
systemctl enable storage
systemctl restart storage
Make sure the service is running:
systemctl status storage
Test by reboot the system.
That's it, now the /mnt/storage will always available.
Benefits of using cached rclone
Avoiding deadlock
When doing an I/O mechanism, you can accidentally running asyncoperation. If it runs for a long time, this could lead to CPU core deadlock. Which can result in system freeze.
By using rclone method, the I/O operation will be completed in shortest time possible.Minimize system resource
Uploading using script (PHP, python, etc) can eat many system resource. And if something happen in the middle of transfer, you have to upload it again from start.
By using rclone method, you can paste and forget. Let it do the thing in the background with minimal resource.Faster download & upload speed
By default, rclone can upload up to 4 files in parallel, and each file will be transferred in 4 connection streams (you can imagine it like using IDM or UGet)
What this method is NOT doing
This method will not magically teleport your files. It may feel instant, but the real upload and download still happen in the background.
Affiliate Link
Forgive me for these affiliate links, but I have to feed my little VPSes at home 
Please don't let them starve. help me feed them some dollars by clicking this link:
Clouvider
2CPU, 4GB RAM, 80GB NVMe, 5TB @ 10Gbit/s : £5 GBP / Month
Use this voucher to get a 10% discount: ADMCQ9GTG90T
HostBrr
A 500GB storagebox which cost €7 eur per YEAR!
Comments
Okhay
Bookmarking this thread. Thanks
So, if i have more space nvme as main disk, can i set
--vfs-cache-max-size 10Gmore than 10G, yes?Can I turn the sullen feelings of a cheap, slow local HDD with reallocated sectors into those of a brand new NVMe?
Sometimes opening emacs feels like a coffe break.
Great job. I use rclone to watch videos from a remote mounted share with 80ms ping. Only it with the cache settings was enough to watch 1080p MKV without interruptions. None of SSHFS, Samba, NFS could do that.
A tiny cheap NVMe dmca ignored VPS + the HostBRR 500GB would make for an interesting seedbox with this method
Hmmm
@sh97 @FatGrizzly what do you think
Correct!
You can also up the
--dir-cache-time to 72hif you want a longer cache age.I can feel your pain, let's cry together
Thank you! You're lucky to get one with 80ms ping!
I'm also surprised when I found out that rclone can be used to watch video.
This seems like perfect privacy protection, and it's very affordable too!
Another way would be to run seedbox on the large storage VPS, while using the DMCA-free one for a VPN tunnel. It is the traditional way and feels like it could be more performant and less complex.
At that point I’d just use a VPN instead. My way provides caching. Not sure how much it would help however. Probably a fair bit for recently downloaded stuff depending on how it’s configured.
Download speeds would likely be improved.
Actually its not real speed after all.
Even with cache but the cache are on same disk. Its just gave an effect on "if dd" but not give any real perfomance. You still have HDD Speed and not nvme in real.
Did you read OPs second sentence?
So this is fake happiness, like diet Coke...
but still a good thought
Diet Coke is life.
thanks for the tutorial. i have just started using rclone as hostbrr DA US is slow on SSHFS.
mine was very basic mount with full cache but no cache expiry and max cache storage.
what i experience now is high IOwait (98.0) after sometime, with no CPU load, read/write operation, no network activity. probably in kilobytes while the vps is just working normally not accessing DA storage.
did you or anyone have a fix for this? or somehow defining cache expiring and max storage solves it?
i think eventually rclone is the way to go as its most resource efficient but SSHFS does things really well for beginners like me
rclone fuse mount implementation lacks some POSIX support like symlinks and hardlinks, which means quite a few applications wouldn't work at all or correctly on the mounted directory.
What is real in this life anyway
I think it is normal.
you are using network storage, depending on your internet speeds, you get IOWait since it's not local storage.
Does the IOwait decrease after a while?
We're all beginners at something
By knowing how mounting works, you're already better than 99.99% of network experts out there.
Open vi instead.
In BusyBox-based installation, it's the same executable as bash, so that it's already resident in memory and does not need to be loaded.
Indeed, rclone is great.
Incidentally, if you require higher performance, try Juciefs, which is block-based unlike Rclone's caching which uses files as granularity. So you'll have a better experience in terms of hits. You can also use Redis as a database to improve retrieval performance when the number of files is particularly high.
Do you have some example commands that we can use with a storage box (via ftp/sftp)?
nice tutorial
thanks
Think you
No, I think Juicefs only supports backends like S3 and Webdav, FTP is not viable.
juicefs support sftp according to official website.
Vi is like an automatic car to me: once inside of it I have no idea what to do of it but to get out.
nice skill !
Use any private trackers and any regular vps would work.
Monthly bandiwith limit will be bottleneck (2.5 TB in this case)