[Tutorial] Turn a cheap HDD storage VPS into NVMe speed

Void

@itachikonoha said:

@Void said:

@emgh said:
A tiny cheap NVMe dmca ignored VPS + the HostBRR 500GB would make for an interesting seedbox with this method

Hmmm

Use any private trackers and any regular vps would work.

Monthly bandiwith limit will be bottleneck (2.5 TB in this case)

For a 500 GB box, it’s more than enough

cybertech

@Jokull said:

@cybertech said:
thanks for the tutorial. i have just started using rclone as hostbrr DA US is slow on SSHFS.

mine was very basic mount with full cache but no cache expiry and max cache storage.

what i experience now is high IOwait (98.0) after sometime, with no CPU load, read/write operation, no network activity. probably in kilobytes while the vps is just working normally not accessing DA storage.

did you or anyone have a fix for this? or somehow defining cache expiring and max storage solves it?

i think eventually rclone is the way to go as its most resource efficient but SSHFS does things really well for beginners like me

I think it is normal.
you are using network storage, depending on your internet speeds, you get IOWait since it's not local storage.
Does the IOwait decrease after a while?

We're all beginners at something
By knowing how mounting works, you're already better than 99.99% of network experts out there.

it has been 98 for days, thing is its just idling now.. and its only 15ms away (Utah > SJC)

if you dont have this issue then I'll try it with cache expiry

Jokull

@davide said:
Vi is like an automatic car to me: once inside of it I have no idea what to do of it but to get out.

And I heard, some owners complained because they can't open the door via voice command.

Jokull

@cybertech said:

it has been 98 for days, thing is its just idling now.. and its only 15ms away (Utah > SJC)

if you dont have this issue then I'll try it with cache expiry

Seems suspicious, you can post your rclone command so we can analyze it if you want.
But, it also might not be related to rclone at all, try disable/close it and see if there's any improvement.

ariq01

I used fuse and got error
Fatal error: mount not ready when mounting remote server via SFTP

then i tried to install fuse3 instead.. Successfully connected.

Jokull

@ariq01 said:
I used fuse and got error
Fatal error: mount not ready when mounting remote server via SFTP

then i tried to install fuse3 instead.. Successfully connected.

I also get similar error yesterday, I think this is because the unmount is still in process.
but maybe your solution is the correct one.

Thank you for the report anyway, I hope this can help a random stranger with trouble shooting an error one day

ariq01

Anyway I also tested with 2 mount point, one with nfs, one with rclone sshfs cache, thx

tailscale nfs

root@redroc:/mnt/redroc# dd if=/dev/zero of=./testfile bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 11.281 s, 9.3 MB/s

tailscale rclone sshfs cache

root@redroc:/opt/plexmedia# dd if=/dev/zero of=./testfile bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 0.510213 s, 206 MB/s

twain

@vicaya said:
rclone fuse mount implementation lacks some POSIX support like symlinks and hardlinks, which means quite a few applications wouldn't work at all or correctly on the mounted directory.

If the mount is used for the storage for a docker container, then these issues would be abstracted away/not present, correct? (unless the docker process itself wants to do linking I guess...)

ralf

@davide said:
Sometimes opening emacs feels like a coffe break.

Remember when people used to joke that emacs was an acronym of "Eight Megs And Continually Swapping". That much RAM would be considered a tiny cache nowadays!

vicaya

@twain said:

@vicaya said:
rclone fuse mount implementation lacks some POSIX support like symlinks and hardlinks, which means quite a few applications wouldn't work at all or correctly on the mounted directory.

If the mount is used for the storage for a docker container, then these issues would be abstracted away/not present, correct? (unless the docker process itself wants to do linking I guess...)

Nope. Docker -v stuff is just a bind mount of a host mount. All fs requests would eventually go through the rclone fuse server, which doesn't support symlinks/hardlinks.

Jokull

@ariq01 said:
Anyway I also tested with 2 mount point, one with nfs, one with rclone sshfs cache, thx

Amazing! Thank you for the comparison!

@twain said:

@vicaya said:
rclone fuse mount implementation lacks some POSIX support like symlinks and hardlinks, which means quite a few applications wouldn't work at all or correctly on the mounted directory.

If the mount is used for the storage for a docker container, then these issues would be abstracted away/not present, correct? (unless the docker process itself wants to do linking I guess...)

@twain: this seems like an advance usage, please try it and report to us the result xD

Regarding the POSIX operation, in my opinion, this method will be mainly used for the backup server.
And copying links to the backup server is not a safe approach at all.

Pilzbaum

I am going to add some information from my side as well, just to give some additional insights I have encountered. For this you'll get some rclone mount parameters I found out to be useful for my usecase (some big and some small files, mainly only read caching, fast access of the beginning of every file, fast updates of the directory structure). I believe that on low bandwidth links or if many more file accesses occur, you might need to recalculate values for your usecase. Also, this is still an experiment I am conducting so I am open to other thoughts/calculations as well!

# This rescans the directory structure regularly for inode changes, this is just to allow quick updates/changes regarding `ls` for instance

dir-cache-time 3s 

# This is the amount of RAM caching that is used for EACH FILE
# I used 1M to allow (full) caching in RAM (not STORAGE) of small files, 
# which speeds up accessing many small files drastically. Be careful that
# too many files fill up you ram (5000 files of 1M =~ 5GB), the effect is not
# very useful (and not intended) for large files, so note that!

buffer-size 1M \

# I set this to full caching (aka write + read, as I require read caching and
# everything else is just write caching)

vfs-cache-mode full \

# This allows cache to be valid for up to ~10 days, I'll come back to it later on why

vfs-cache-max-age 250h \

# This is the absolut amount of maximum cache to use, I have this amount
# of storage available, you may check how much you can use for caching.

vfs-cache-max-size 150G \

# This enforces that even if 150G are not filled, at least 16GB are kept free
# to the OS or other services.

vfs-cache-min-free-space 16G \

# This sets the base chunk size of a file to 25MB, which I believe is nice as
# the number of request per file gets reduced on the remote server
# (just what I believe it to be, it works nice for me at least, I might reduce it maybe)

vfs-read-chunk-size 25M \

# This allows the doubling of chunks to be limited to 100M in total, this
# might be a tad high. Remember that every chunk will be accessed, so
# if this value is too high, the link will be saturated for a long time even
# if no file is accessed anymore

vfs-read-chunk-size-limit 100M \

# This allows to detect inode changes faster (but may be less secure)

vfs-fast-fingerprint \

# Now this is the kinda the magic number the caching is based on. On
# every file access, the first 50M of a file are downloaded as well and
# stored on the STORAGE (not RAM). I picked this number, as for media
# files (depending on the bitrate, lets assume 10Mbps) this stores around
# 40 seconds worth of content on the disk. This is (for me) a decent
# amount of data that (pre-)transcoding or light skipping of 5-10 seconds
# kinda appears to be accessing the file instant, as the data is stored on
# the disk this way.

vfs-read-ahead 50M \

Now, next to being able to cache, I also want to actually cache files for fast access. For this, I wrote a bash script that accesses the first few MB of a file, which are stored on the disk/ram due to the caching with rclone. This allows to access files from the beginning to appear to be read instantaneously afterwards, so with local disk speed.
Some thoughts:
Be CAREFUL, this puts up an enormous amount of load onto your server and on the remote when executed, because every file will be rescanned and checked (with rclone mainly using checksums md5/sha256!) and possibly downloaded as well.
This was the reason I put up 10 days of caching validity, as I mainly read static files which do not change anymore. Also be careful that you might use up your bandwidth really fast when caching very often. Imagine 5000 files and caching 50M each => 250GB storage/bandwidth! Do this 10 times a month (every 3 days) and you have used upto 2.5TB just for caching 40 seconds of data for each file Just don't do this, it really adds so much stress to the system(s) (especially on shared systems, be nice to your neighbors!)

#!/bin/bash

# Parameters
MOUNT="/mnt/hostbrrde"
WORKERS=1 # Future compatibility, be careful tho, might hang your system or the remote if using many!
SUBDIRS=("TV" "Movies")  # Add more subdirectories as needed, I specifically do not want to cache every directory of my remote mount

# Function to access file
process_file() {
    local file="$1"
    local CHUNK_SIZE="50M"
    # Check if the file exists to debug potential issues
    if [[ -f "$file" ]]; then
        dd if="$file" bs=${CHUNK_SIZE} count=1 of=/dev/null 2>/dev/null
    fi
}

export -f process_file  # Export the function for parallel use

# Process each subdirectory in parallel
for subdir in "${SUBDIRS[@]}"; do
    find "${MOUNT}/${subdir}/" -type f -print0 | \
    parallel --null -j${WORKERS} process_file
done

Here some benches (two consecutive runs, first without cache, second when re-accessing and with RAM and STORAGE cache intact), when I sadly noticed network performance is shit atm :P. Might need to investigate

dd if=1MB.file of=/dev/zero status=progress
1032704 bytes (1.0 MB, 1008 KiB) copied, 1 s, 969 kB/s
2048+0 records in
2048+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 1.0662 s, 983 kB/s

dd if=1MB.file of=/dev/zero status=progress   
2048+0 records in
2048+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0034107 s, 307 MB/s


dd if=16MB.file of=/dev/zero status=progress  
16761344 bytes (17 MB, 16 MiB) copied, 15 s, 1.1 MB/s 
32768+0 records in
32768+0 records out
16777216 bytes (17 MB, 16 MiB) copied, 15.1521 s, 1.1 MB/s

dd if=16MB.file of=/dev/zero status=progress
32768+0 records in
32768+0 records out
16777216 bytes (17 MB, 16 MiB) copied, 0.0486071 s, 345 MB/s


dd if=100MB.file of=/dev/zero status=progress
103793152 bytes (104 MB, 99 MiB) copied, 101 s, 1.0 MB/s
204800+0 records in
204800+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 101.319 s, 1.0 MB/s

dd if=100MB.file of=/dev/zero status=progress
204800+0 records in
204800+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 0.333842 s, 314 MB/s

hostnoob

@Void said:

@itachikonoha said:

@Void said:

@emgh said:
A tiny cheap NVMe dmca ignored VPS + the HostBRR 500GB would make for an interesting seedbox with this method

Hmmm

Use any private trackers and any regular vps would work.

Monthly bandiwith limit will be bottleneck (2.5 TB in this case)

For a 500 GB box, it’s more than enough

Especially for private trackers. You're not going to be hitting high ratios on a decent tracker

twain

@vicaya said:
rclone fuse mount implementation lacks some POSIX support like symlinks and hardlinks, which means quite a few applications wouldn't work at all or correctly on the mounted directory.

If the mount is used for the storage for a docker container, then these issues would be abstracted away/not present, correct? (unless the docker process itself wants to do linking I guess...)

Nope. Docker -v stuff is just a bind mount of a host mount. All fs requests would eventually go through the rclone fuse server, which doesn't support symlinks/hardlinks.

Hmm, ok. But isn't there an rclone docker volume plugin? Would things behave using that w regard to links?

vicaya

@twain said:

@vicaya said:
rclone fuse mount implementation lacks some POSIX support like symlinks and hardlinks, which means quite a few applications wouldn't work at all or correctly on the mounted directory.

If the mount is used for the storage for a docker container, then these issues would be abstracted away/not present, correct? (unless the docker process itself wants to do linking I guess...)

Nope. Docker -v stuff is just a bind mount of a host mount. All fs requests would eventually go through the rclone fuse server, which doesn't support symlinks/hardlinks.

Hmm, ok. But isn't there an rclone docker volume plugin? Would things behave using that w regard to links?

The volume plugin just make things a bit easier. The underlying implementation is the same, which doesn't work with links correctly. You can test it easily by typing ln foo bar inside the container.

ariq01

I faced a problem.. my plex can't read the video after mounted. I tried to trigger scanning library, but still didn't work..

Void

@hostnoob said:

@Void said:

@itachikonoha said:

@Void said:

@emgh said:
A tiny cheap NVMe dmca ignored VPS + the HostBRR 500GB would make for an interesting seedbox with this method

Hmmm

Use any private trackers and any regular vps would work.

Monthly bandiwith limit will be bottleneck (2.5 TB in this case)

For a 500 GB box, it’s more than enough

Especially for private trackers. You're not going to be hitting high ratios on a decent tracker

Correct. Besides, the goal is not to race for high uploads and ratios but just to be safe from all copyright crap.

Jokull

@ariq01 said:
I faced a problem.. my plex can't read the video after mounted. I tried to trigger scanning library, but still didn't work..

Maybe try using another video player? Celluloid plays video just fine on my computer.
If it still doesn't work, you can post your configuration here if you want

Jokull

@Pilzbaum said:
I am going to add some information from my side as well, just to give some additional insights I have encountered. For this you'll get some rclone mount parameters I found out to be useful for my usecase (some big and some small files, mainly only read caching, fast access of the beginning of every file, fast updates of the directory structure). I believe that on low bandwidth links or if many more file accesses occur, you might need to recalculate values for your usecase. Also, this is still an experiment I am conducting so I am open to other thoughts/calculations as well!

Thank you very much for the insight!
Your description in the first snippet is especially useful, it helped me understand many options that I didn't even know.

I really hope that the official docs are as easy to understand as your description

TimboJones

@davide said:

@rm_ said:
Great job. I use rclone to watch videos from a remote mounted share with 80ms ping. Only it with the cache settings was enough to watch 1080p MKV without interruptions. None of SSHFS, Samba, NFS could do that.

@yoursunny said:

@davide said:
Sometimes opening emacs feels like a coffe break.

Open vi instead.
In BusyBox-based installation, it's the same executable as bash, so that it's already resident in memory and does not need to be loaded.

Vi is like an automatic car to me: once inside of it I have no idea what to do of it but to get out.

Dafuq? There's less to do than in a standard.

davide

@TimboJones said:
Dafuq? There's less to do than in a standard.

hit me

Jokull

@danblaze said:

@BasToTheMax said:

@danblaze said:
Indeed, rclone is great.

Incidentally, if you require higher performance, try Juciefs, which is block-based unlike Rclone's caching which uses files as granularity. So you'll have a better experience in terms of hits. You can also use Redis as a database to improve retrieval performance when the number of files is particularly high.

Do you have some example commands that we can use with a storage box (via ftp/sftp)?

No, I think Juicefs only supports backends like S3 and Webdav, FTP is not viable.

Correct.
JuiceFS can't be used as a normal SFTP mount, it is different from rclone

First of all, thank you to @danblaze for letting me know about JuiceFS.
Here is my report.

I have installed and tested JuiceFS:

• On JuiceFS, the data will be encrypted, and you need to install it to view the content.
• They didn't mention it or put an alert for it, but your target dir will be rewriten and formated!, all data will be lost.
• I tried to cancel the write operation via terminal, but it keep spawning and resuming the transfer. I have to wait it complete the write, and only by then the process can be closed.
• Unlike rclone, it doesn't feel like NVMe to me. Because I have to wait for the copy dialog for a long time, the transfer process is in the foreground, not in the background.

Anyway, here is the transfer speed comparison using cbm:

• JuiceFS

Interface      Receive            Transmit        Total
xxxxxxx        71.48 kB/s         3.32 MB/s       3.40 MB/s

• rclone

Interface      Receive            Transmit        Total
xxxxxxx        207.91 kB/s        9.17 MB/s       9.38 MB/s

Both is run with default configuration.
The normal speed with SFTP is about 1.8 Mb/s, so we get like 2x transfer speed increase with JuiceFS. But still much lower than rclone, with 5x speed increase.

A litte mistake

I was doing a small test, but got my remote dir nuked instead. sigh, need to upload again from the start.

JuiceFS should at least put a warning in their docs. I know the juicefs format is looks suspicious, but their documentation is not clear, I was assuming it will format my empty local dir.

The funny thing is:

When searching on google on how to configure it on SFTP, the best result that I can find is this thread.
So, come on guys, share your knowledge! Your little snippet might help a random person troubleshoot their problem

0xC7

I have tried rclone served as webdav or sftp. However (probably) don't have any 'good' vps_vm network resource to compare.
Could anyone here have test this method with AlistGo ?

itachikonoha

@Void said:

@itachikonoha said:

@Void said:

@emgh said:
A tiny cheap NVMe dmca ignored VPS + the HostBRR 500GB would make for an interesting seedbox with this method

Hmmm

Use any private trackers and any regular vps would work.

Monthly bandiwith limit will be bottleneck (2.5 TB in this case)

For a 500 GB box, it’s more than enough

It isn't the same in my trackers. When you download a new torrent with say 3 or 4 seeds, later the upload for that only file (20gb ish) may cross 60-70 GB in just 2-3 days.

So 2.5 TB becomes very less in these cases.

Void

@itachikonoha said:

@Void said:

@itachikonoha said:

@Void said:

@emgh said:
A tiny cheap NVMe dmca ignored VPS + the HostBRR 500GB would make for an interesting seedbox with this method

Hmmm

Use any private trackers and any regular vps would work.

Monthly bandiwith limit will be bottleneck (2.5 TB in this case)

For a 500 GB box, it’s more than enough

It isn't the same in my trackers. When you download a new torrent with say 3 or 4 seeds, later the upload for that only file (20gb ish) may cross 60-70 GB in just 2-3 days.

So 2.5 TB becomes very less in these cases.

Most likely, if you grab it the moment it’s uploaded. But you can simply stop seeding once it reaches a 1:1 ratio. Most trackers require seeding for a certain number of days or until you reach a 1:1 ratio, so pick those that have the latter.

Jokull

@0xC7 said:
I have tried rclone served as webdav or sftp. However (probably) don't have any 'good' vps_vm network resource to compare.
Could anyone here have test this method with AlistGo ?

Nah, you don't need to compare.
If you feel like your current setup can satisfy your soul, then use it.
Trust your heart

Can't find the docs for AlistGo, their site is also 502.

fuqet

Will this work on windows server?

Just295

@fuqet There is no reason this shouldn't. https://rclone.org/downloads/ There is a Windows installer, but I have never used this. I can say and I have used several times, MountainDuck.io Its a onetime paid software $35 and it will allow you to connect to a boat load of connections like sFTP/WebDav/S3/SSH and so forth, and then mount them as a drive letter. There is a 14 day trial if you wanted to test it out.

Thanks,
Anthony

Jokull

@fuqet said:
Will this work on windows server?

Hi, from the documentation, rclone supported windows with minimum version: Windows 10, Windows Server 2016

You can check https://rclone.org/install/#windows-precompiled for further instruction.

Andreix

@fuqet said:
Will this work on windows server?

This article should do the trick: Advanced Remote Storage Optimization for Windows: Making Remote Storage Feel Like Local NVMe

PS: Feel free to embed/redistribute/cite.