All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Optimizing rclone SFTP to replace SSHFS
PineappleM
Member
Since SSHFS is deprecated, the most commonly promoted alternative is rclone. I followed the rclone tutorial posted here to get started.
I'm testing my rclone configuration on two Debian 12 virtual machines on my PC. Both systems were installed from the ISO on the official Debian website, are on the same SSD, and are on the same LAN. With this setup, I want to configure rclone to run as optimally as possible so that any slowdown on production machines can be attributed to their network connections.
Despite having set up a cache, dd is reporting much lower speeds on the rclone directory than on the local file system. Since artificial benchmarks don't always reflect real world cases, I copied a NodeJS app folder with a bunch of tiny files and it was also slow. (Benchmark results are at the bottom of the post.)
Here are the commands I ran to set up rclone. I ran everything as root and put all the config files in the system root (/) directory for quick and dirty testing. Since these are VMs, I can roll back snapshots to test any modifications in the installation or setup process.
# apt-get update
# apt-get install rclone fuse
# mkdir /mnt/storage
# mkdir /mnt/.cache
# ssh-keygen -t ed25519 -C "storage-sftp" -f "/test_key" -N "" > /dev/null
# chmod 600 /test_key
I added the generated public key to the "storage" VM. Then I created rclone.conf with the following contents:
[storage-sftp]
type = sftp
host = 192.168.1.189
port = 22
user = pineapple
key_file = /test_key
Finally I ran rclone with this command:
# rclone mount storage-sftp:/home /mnt/storage --config /rclone.conf --dir-cache-time 72h --cache-dir=/mnt/.cache --vfs-cache-mode full --vfs-cache-max-size 10G
I tested both dd and cp on the rclone directory and a local directory, and these are the summarized results:
dd if=/dev/zero of=./testfile bs=10K count=1000
Local directory: 248 MB/s
rclone directory: 27.8 MB/s
NodeJS folder copy: 4103 files totaling 125 MB
Local directory: 0.6-0.7 s
rclone directory: 14-15 s
Is this as good as rclone gets, or am I doing something wrong or forgot something that would make it faster? If this is the best that I can expect from rclone then I'll push this to production, but if this can be further improved I'm happy to hear suggestions.
Hopefully this helps future readers with setting up their own rclone mounts over SFTP. Thanks!
Comments
https://rclone.org/commands/rclone_mount/#vfs-performance
These two knobs might help you, but I wouldn't expect same performance as local folder
My small experience in the life :
SSHFS was never a good solution for me to save because of other side of tunnel, all external mount can cause trouble.
I didn't need disk backup, just to save only important data folders.
I had performance issues with RCLONE, then i moved to LFTP which can sync too.
LFTP easy to do !
@PineappleM
What's your core point(s)? Encrypted transport and/or encrypted storage on target server and/or encrypted connection/session? Different things with different consequences for speed.
Btw. SSHFS never was the best (or even just a good) solution IMO.
Obviously depends on the use case, but that’s not too bad. We’re all so used to high write numbers many of us forget how little we actually need. But if your use case does need more, and I guess you do as I’m obviously not suggesting you don’t know what you need, I feel like maybe the whole concept is flawed. Is there any way you can convert this data to be stored in a remote database that each box connects to?
This is the kind of thing where AI as a brainstorming buddy really shines. Just because it generally has a deeper reference pool.
This is my speed writing from my local PC with NVMe disk to a remote ProFTPd SFTP rclone mount with HDD:
my rclone mount flags are like this:
The remote is a Dedi not a Virtual Server.
Try WebDAV instead of SFTP. Near wireline speeds then. WebDAV server built into rclone aswell.
try LFTP...
a few months back I would suggest minio, but after their recent changes I am also migrating away. Tried setup a seaweedFS deployment on a host-c storage vps, reasonably light memory footprint, and this thing advertises to work well with large amount of tiny files.
not a hardcore benchmark guy so subjectively it worked fine, at least better than sftp xD
if you don't mind, please try too JuiceFS.
Was it easy to setup?
wait.....why is sshfs deprecated? I'm relying on it lol...should I change?
One thing Rclone can't do for me but sshfs allows the host to change ownership (user/group) of the directories/files on the receiving end, as running docker container often sensitive to ownership so I can only depend on sshfs (compression=no to improve speed). Would be delighted to try out better alternatives!
Bookmarked!
depends on skill level ofc. they have a compose file, so you can easily compose up and test, but there are still s3 bucket reverse proxying, grpc certs generating, and a few optimization flags to set, took me 2-3 hours to debug and finalize my local mount, ai helped alot though.