[Tutorial] Turn a cheap HDD storage VPS into NVMe speed

twain

I'm assuming one of the use cases for rclone is to use for/with Nextcloud?

Here is an alternative to Nextcloud called Filestash where using sftp is an option for the storage, so you could take rclone out of the equation if this sort of application your use case.

I haven't messed with it yet, but looks pretty cool, although the config options looks like it could be messy/complicated.

https://github.com/mickael-kerjean/filestash

FunFap

Damn tutor, auto bookmark.

mickaelkerjean

Filestash author here, happy to answer any question about my software

laey

@Jokull said:
port = [Find this: You must use port ***** for SSH]

I tried before, but I got stuck when connecting to SFTP using port 2222, as the connection was rejected. Where can I find the port configuration?

ModMod

Thanks for the tutorial

laey

@laey said:

I tried before, but I got stuck when connecting to SFTP using port 2222, as the connection was rejected. Where can I find the port configuration?

Now I've figured it out, Hostbrr always uses port 53211 for that. Thank you.

root@redacted:~# dd if=/dev/zero of=/mnt/storagebox-hostbrr/testfile bs=1M count=1000 
1048576000 bytes (1.0 GB, 1000 MiB) copied, 97.4623 s, 10.8 MB/s

root@redacted:~# dd if=/dev/zero of=/mnt/storagebox-hostbrr/testfile bs=1M count=1000
1048576000 bytes (1.0 GB, 1000 MiB) copied, 2.7383 s, 383 MB/s

Grey_01

Is there any way to use this method for SSD to cache for local hdd disk, not the remote storage

aRNoLD

HostBrr soon into out of stock again.

mikrocybercloud

Awesome

razorblade

Bookmark saved.

vikas111981

@Grey_01 said:
Is there any way to use this method for SSD to cache for local hdd disk, not the remote storage

I have this very same question. I went through the details on official site and understand that it would mount remote storage and S3 but not local. I tried solutions like bcache for VPS but they are not very easy to implement. Tried this, it's very easy but I have VPS with NVME and HDD both and want to use HDD as main storage cached through NVME on same VPS but don't know how

e2bs2k1

This method is actually documented(in another way) in rclone usage manual. I'm suprised that people see this fresh.

dav848

How about docker, Right Now I'm running docker on hdd

ScreenReader

necromancer here, neat thread. I managed to use my 1TB hostbrr storage for garage s3.

overall setup;

rclone config

[hostbrr-sftp]
type = sftp
host = 192.200.x.x
user = username
port = 53211 
key_pem = -----BEGIN OPENSSH PRIVATE KEY-----\nb3BlbnNz...YS1qcA==\n-----END OPENSSH PRIVATE KEY-----\n
use_insecure_cipher = false
shell_type = unix
md5sum_command = md5sum
sha1sum_command = sha1sum

[hostbrr-crypt]
type = crypt
remote = hostbrr-sftp:/home/username/path/to/folder
password = secret
password2 = secret

---
services:
  rclone-mount:
    image: rclone/rclone:latest
    command: >
      mount hostbrr-crypt: /storage
      --allow-other
      --allow-non-empty
      --no-modtime
      --vfs-cache-mode writes
      --vfs-cache-max-size 5G
      --dir-cache-time 1h
      --daemon-timeout 1h
    volumes:
      - /home/docker/.config/rclone:/config/rclone:ro
      - /mnt/remote-storage:/storage:rshared
    cap_add:
      - SYS_ADMIN
    devices:
      - /dev/fuse
    security_opt:
      - apparmor:unconfined
    restart: unless-stopped
    healthcheck:
      test: ["CMD-SHELL", "mountpoint -q /storage"]
      interval: 5s
      timeout: 3s
      retries: 20

  garage:
    image: dxflrs/garage:v2.3.0
    restart: unless-stopped
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.garage-s3.entrypoints=http"
      - "traefik.http.routers.garage-s3.rule=Host(`domain.name`)"
      - "traefik.http.middlewares.garage-s3-https-redirect.redirectscheme.scheme=https"
      - "traefik.http.routers.garage-s3.middlewares=garage-s3-https-redirect"
      - "traefik.http.routers.garage-s3-secure.entrypoints=https"
      - "traefik.http.routers.garage-s3-secure.rule=Host(`domain.name`)"
      - "traefik.http.routers.garage-s3-secure.tls=true"
      - "traefik.http.routers.garage-s3-secure.service=garage-s3"
      - "traefik.http.services.garage-s3.loadbalancer.server.port=3900"
      - "traefik.docker.network=proxy"
    volumes:
      - ./garage.toml:/etc/garage.toml
      - ./meta:/var/lib/garage/meta
      - /mnt/remote-storage:/var/lib/garage/data:rshared
    networks:
      - external
    depends_on:
      rclone-mount:
        condition: service_healthy

networks:
  external:
    external: true
    name: proxy

since this is rshared, make sure to set it up.

mount --make-rshared /mnt/remote-storage

the garage.toml just basic following https://garagehq.deuxfleurs.fr/documentation/quick-start/

metadata_dir = "/var/lib/garage/meta"
data_dir = "/var/lib/garage/data"
db_engine = "sqlite"

replication_factor = 1

rpc_bind_addr = "[::]:3901"
rpc_public_addr = "127.0.0.1:3901"
rpc_secret = "changeme"

[s3_api]
s3_region = "garage"
api_bind_addr = "[::]:3900"
root_domain = "s3.garage.localhost"

[s3_web]
bind_addr = "[::]:3902"
root_domain = "web.garage.localhost"
index = "index.html"

[admin]
api_bind_addr = "[::]:3903"
admin_token = "changeme"
metrics_token = "changeme"

the meta folder probably not safe being stored remotely so i put it in local. 15GB ssd vps being used 5gb for cache is still plenty to store the meta folder.

in this setup at very least i can confirm the disk usage in hostbrr is increasing whenever i upload something, so that proves the storage is no longer in local vps disk. might need to adjust --vfs-cache-mode but i havent tested it any further