Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


VULTR Seattle Node Failure - Page 3
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

VULTR Seattle Node Failure

13»

Comments

  • I'll be surprised if any of the bigger brands such as Vultr go to Ryzen in HF VPS, they'll either stick with Intel or perhaps go to EPYC at a push. But compare the cost of some of the higher frequency AMD EPYCs vs Intel Es for example, Intels are the better option there.

  • jlayjlay Member
    edited June 2020

    Not going to get the vendor agreements you want as a large cloud with Ryzen systems. It's mostly enterprise class gear that comes with things like service contracts at that scale

    You can't afford hands to spend hours working on a downed node. Call Dell or someone and they'll take a new one off the pallet for you.

    Some places even work by the rack. Just using available capacity and live migration to eventually replace multiple systems at once.

  • raindog308raindog308 Administrator, Veteran

    @Brend4n said: I mean, providers that have network/redundant/CEPH storage are pretty comparable in pricing these days... UpCloud and Heficed come to mind. Heficed use CEPH with triple replication, and UpCloud their own inhouse I believe.

    What's the comparison on that vs. local SSD/NVME though?

    I think most people would prefer local RAID for performance reasons.

    Thanked by 1WSCallum
  • Brend4nBrend4n Member
    edited June 2020

    @raindog308 said:

    @Brend4n said: I mean, providers that have network/redundant/CEPH storage are pretty comparable in pricing these days... UpCloud and Heficed come to mind. Heficed use CEPH with triple replication, and UpCloud their own inhouse I believe.

    What's the comparison on that vs. local SSD/NVME though?

    I think most people would prefer local RAID for performance reasons.

    Here are some tests I did with various providers. Speeds should be more than sufficient for most people, only really beat by NVMe. I'd take storage reliability any day, esp for production.

    Vultr HF (Local NVMe. No RAID)

    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    #              Yet-Another-Bench-Script              #
    #                     v2020-06-20                    #
    # https://github.com/masonr/yet-another-bench-script #
    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    
    Wed Jun 24 18:15:14 UTC 2020
    
    Basic System Information:
    ---------------------------------
    Processor  : Intel Core Processor (Skylake, IBRS)
    CPU cores  : 1 @ 3792.000 MHz
    AES-NI     : ✔ Enabled
    VM-x/AMD-V : ❌ Disabled
    RAM        : 1.9Gi
    Swap       : 0B
    Disk       : 63G
    
    fio Disk Speed Tests (Mixed R/W 50/50):
    ---------------------------------
    Block Size | 4k            (IOPS) | 64k           (IOPS)
      ------   | ---            ----  | ----           ----
    Read       | 248.47 MB/s  (62.1k) | 638.28 MB/s   (9.9k)
    Write      | 249.12 MB/s  (62.2k) | 641.64 MB/s  (10.0k)
    Total      | 497.60 MB/s (124.4k) | 1.27 GB/s    (19.9k)
               |                      |
    Block Size | 512k          (IOPS) | 1m            (IOPS)
      ------   | ---            ----  | ----           ----
    Read       | 701.29 MB/s   (1.3k) | 726.14 MB/s    (709)
    Write      | 738.55 MB/s   (1.4k) | 774.50 MB/s    (756)
    Total      | 1.43 GB/s     (2.8k) | 1.50 GB/s     (1.4k)
    

    BuyVM (Local NVMe)

    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    #              Yet-Another-Bench-Script              #
    #                     v2020-06-20                    #
    # https://github.com/masonr/yet-another-bench-script #
    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    
    Wed 24 Jun 2020 11:28:04 AM PDT
    
    Basic System Information:
    ---------------------------------
    Processor  : AMD Ryzen 9 3900X 12-Core Processor
    CPU cores  : 1 @ 3792.872 MHz
    AES-NI     : ✔ Enabled
    VM-x/AMD-V : ✔ Enabled
    RAM        : 987Mi
    Swap       : 1.0Gi
    Disk       : 270G
    
    fio Disk Speed Tests (Mixed R/W 50/50):
    ---------------------------------
    Block Size | 4k            (IOPS) | 64k           (IOPS)
      ------   | ---            ----  | ----           ----
    Read       | 292.83 MB/s  (73.2k) | 2.44 GB/s    (38.2k)
    Write      | 293.61 MB/s  (73.4k) | 2.46 GB/s    (38.4k)
    Total      | 586.45 MB/s (146.6k) | 4.91 GB/s    (76.7k)
               |                      |
    Block Size | 512k          (IOPS) | 1m            (IOPS)
      ------   | ---            ----  | ----           ----
    Read       | 2.52 GB/s     (4.9k) | 2.48 GB/s     (2.4k)
    Write      | 2.65 GB/s     (5.1k) | 2.64 GB/s     (2.5k)
    Total      | 5.18 GB/s    (10.1k) | 5.12 GB/s     (5.0k)
    

    UpCloud (NAS, MaxIOPs Inhouse)

    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    #              Yet-Another-Bench-Script              #
    #                     v2020-06-20                    #
    # https://github.com/masonr/yet-another-bench-script #
    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    
    Wed 24 Jun 2020 06:19:20 PM UTC
    
    Basic System Information:
    ---------------------------------
    Processor  : Intel(R) Xeon(R) Gold 6136 CPU @ 3.00GHz
    CPU cores  : 1 @ 2992.968 MHz
    AES-NI     : ✔ Enabled
    VM-x/AMD-V : ❌ Disabled
    RAM        : 1.9Gi
    Swap       : 0B
    Disk       : 50G
    
    fio Disk Speed Tests (Mixed R/W 50/50):
    ---------------------------------
    Block Size | 4k            (IOPS) | 64k           (IOPS)
      ------   | ---            ----  | ----           ----
    Read       | 117.70 MB/s  (29.4k) | 408.28 MB/s   (6.3k)
    Write      | 118.01 MB/s  (29.5k) | 410.43 MB/s   (6.4k)
    Total      | 235.72 MB/s  (58.9k) | 818.71 MB/s  (12.7k)
               |                      |
    Block Size | 512k          (IOPS) | 1m            (IOPS)
      ------   | ---            ----  | ----           ----
    Read       | 391.50 MB/s    (764) | 388.06 MB/s    (378)
    Write      | 412.30 MB/s    (805) | 413.90 MB/s    (404)
    Total      | 803.81 MB/s   (1.5k) | 801.96 MB/s    (782)
    

    Heficed (NAS, CEPH Triple Replication)

    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    #              Yet-Another-Bench-Script              #
    #                     v2020-06-20                    #
    # https://github.com/masonr/yet-another-bench-script #
    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    
    Wed 24 Jun 2020 06:09:08 PM UTC
    
    Basic System Information:
    ---------------------------------
    Processor  : Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz
    CPU cores  : 2 @ 2199.998 MHz
    AES-NI     : ✔ Enabled
    VM-x/AMD-V : ❌ Disabled
    RAM        : 1.9Gi
    Swap       : 0B
    Disk       : 20G
    
    fio Disk Speed Tests (Mixed R/W 50/50):
    ---------------------------------
    Block Size | 4k            (IOPS) | 64k           (IOPS)
      ------   | ---            ----  | ----           ----
    Read       | 33.04 MB/s    (8.2k) | 371.78 MB/s   (5.8k)
    Write      | 33.09 MB/s    (8.2k) | 373.73 MB/s   (5.8k)
    Total      | 66.13 MB/s   (16.5k) | 745.52 MB/s  (11.6k)
               |                      |
    Block Size | 512k          (IOPS) | 1m            (IOPS)
      ------   | ---            ----  | ----           ----
    Read       | 382.63 MB/s    (747) | 355.50 MB/s    (347)
    Write      | 402.96 MB/s    (787) | 379.17 MB/s    (370)
    Total      | 785.59 MB/s   (1.5k) | 734.68 MB/s    (717)
    

    OVH VPS (Local SSD)

    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    #              Yet-Another-Bench-Script              #
    #                     v2020-06-20                    #
    # https://github.com/masonr/yet-another-bench-script #
    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    
    Wed 24 Jun 2020 06:21:04 PM UTC
    
    Basic System Information:
    ---------------------------------
    Processor  : Intel Xeon E3-12xx v2 (Ivy Bridge, IBRS)
    CPU cores  : 1 @ 2599.998 MHz
    AES-NI     : ✔ Enabled
    VM-x/AMD-V : ✔ Enabled
    RAM        : 1.9Gi
    Swap       : 0B
    Disk       : 20G
    
    fio Disk Speed Tests (Mixed R/W 50/50):
    ---------------------------------
    Block Size | 4k            (IOPS) | 64k           (IOPS)
      ------   | ---            ----  | ----           ----
    Read       | 8.43 MB/s     (2.1k) | 34.39 MB/s     (537)
    Write      | 8.47 MB/s     (2.1k) | 34.77 MB/s     (543)
    Total      | 16.90 MB/s    (4.2k) | 69.17 MB/s    (1.0k)
               |                      |
    Block Size | 512k          (IOPS) | 1m            (IOPS)
      ------   | ---            ----  | ----           ----
    Read       | 55.78 MB/s     (108) | 66.54 MB/s      (64)
    Write      | 58.70 MB/s     (114) | 71.04 MB/s      (69)
    Total      | 114.48 MB/s    (222) | 137.59 MB/s    (133)
    

    Personal VM (SSD, no RAID)

    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    #              Yet-Another-Bench-Script              #
    #                     v2020-06-20                    #
    # https://github.com/masonr/yet-another-bench-script #
    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    
    Wed 24 Jun 2020 11:24:54 AM PDT
    
    Basic System Information:
    ---------------------------------
    Processor  : Common KVM processor
    CPU cores  : 4 @ 4007.996 MHz
    AES-NI     : ❌ Disabled
    VM-x/AMD-V : ❌ Disabled
    RAM        : 3.9Gi
    Swap       : 4.0Gi
    Disk       : 46G
    
    fio Disk Speed Tests (Mixed R/W 50/50):
    ---------------------------------
    Block Size | 4k            (IOPS) | 64k           (IOPS)
      ------   | ---            ----  | ----           ----
    Read       | 146.98 MB/s  (36.7k) | 154.68 MB/s   (2.4k)
    Write      | 147.37 MB/s  (36.8k) | 155.49 MB/s   (2.4k)
    Total      | 294.35 MB/s  (73.5k) | 310.18 MB/s   (4.8k)
               |                      |
    Block Size | 512k          (IOPS) | 1m            (IOPS)
      ------   | ---            ----  | ----           ----
    Read       | 208.52 MB/s    (407) | 179.76 MB/s    (175)
    Write      | 219.60 MB/s    (428) | 191.73 MB/s    (187)
    Total      | 428.12 MB/s    (835) | 371.50 MB/s    (362)
    
  • @Brend4n said:

    @raindog308 said:

    @Brend4n said: I mean, providers that have network/redundant/CEPH storage are pretty comparable in pricing these days... UpCloud and Heficed come to mind. Heficed use CEPH with triple replication, and UpCloud their own inhouse I believe.

    What's the comparison on that vs. local SSD/NVME though?

    I think most people would prefer local RAID for performance reasons.

    Here are some tests I did with various providers. Speeds should be more than sufficient for most people, only really beat by NVMe. I'd take storage reliability any day, esp for production.

    Vultr HF (Local NVMe. No RAID)

    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    #              Yet-Another-Bench-Script              #
    #                     v2020-06-20                    #
    # https://github.com/masonr/yet-another-bench-script #
    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    
    Wed Jun 24 18:15:14 UTC 2020
    
    Basic System Information:
    ---------------------------------
    Processor  : Intel Core Processor (Skylake, IBRS)
    CPU cores  : 1 @ 3792.000 MHz
    AES-NI     : ✔ Enabled
    VM-x/AMD-V : ❌ Disabled
    RAM        : 1.9Gi
    Swap       : 0B
    Disk       : 63G
    
    fio Disk Speed Tests (Mixed R/W 50/50):
    ---------------------------------
    Block Size | 4k            (IOPS) | 64k           (IOPS)
      ------   | ---            ----  | ----           ----
    Read       | 248.47 MB/s  (62.1k) | 638.28 MB/s   (9.9k)
    Write      | 249.12 MB/s  (62.2k) | 641.64 MB/s  (10.0k)
    Total      | 497.60 MB/s (124.4k) | 1.27 GB/s    (19.9k)
               |                      |
    Block Size | 512k          (IOPS) | 1m            (IOPS)
      ------   | ---            ----  | ----           ----
    Read       | 701.29 MB/s   (1.3k) | 726.14 MB/s    (709)
    Write      | 738.55 MB/s   (1.4k) | 774.50 MB/s    (756)
    Total      | 1.43 GB/s     (2.8k) | 1.50 GB/s     (1.4k)
    

    BuyVM (Local NVMe)

    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    #              Yet-Another-Bench-Script              #
    #                     v2020-06-20                    #
    # https://github.com/masonr/yet-another-bench-script #
    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    
    Wed 24 Jun 2020 11:28:04 AM PDT
    
    Basic System Information:
    ---------------------------------
    Processor  : AMD Ryzen 9 3900X 12-Core Processor
    CPU cores  : 1 @ 3792.872 MHz
    AES-NI     : ✔ Enabled
    VM-x/AMD-V : ✔ Enabled
    RAM        : 987Mi
    Swap       : 1.0Gi
    Disk       : 270G
    
    fio Disk Speed Tests (Mixed R/W 50/50):
    ---------------------------------
    Block Size | 4k            (IOPS) | 64k           (IOPS)
      ------   | ---            ----  | ----           ----
    Read       | 292.83 MB/s  (73.2k) | 2.44 GB/s    (38.2k)
    Write      | 293.61 MB/s  (73.4k) | 2.46 GB/s    (38.4k)
    Total      | 586.45 MB/s (146.6k) | 4.91 GB/s    (76.7k)
               |                      |
    Block Size | 512k          (IOPS) | 1m            (IOPS)
      ------   | ---            ----  | ----           ----
    Read       | 2.52 GB/s     (4.9k) | 2.48 GB/s     (2.4k)
    Write      | 2.65 GB/s     (5.1k) | 2.64 GB/s     (2.5k)
    Total      | 5.18 GB/s    (10.1k) | 5.12 GB/s     (5.0k)
    

    UpCloud (NAS, MaxIOPs Inhouse)

    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    #              Yet-Another-Bench-Script              #
    #                     v2020-06-20                    #
    # https://github.com/masonr/yet-another-bench-script #
    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    
    Wed 24 Jun 2020 06:19:20 PM UTC
    
    Basic System Information:
    ---------------------------------
    Processor  : Intel(R) Xeon(R) Gold 6136 CPU @ 3.00GHz
    CPU cores  : 1 @ 2992.968 MHz
    AES-NI     : ✔ Enabled
    VM-x/AMD-V : ❌ Disabled
    RAM        : 1.9Gi
    Swap       : 0B
    Disk       : 50G
    
    fio Disk Speed Tests (Mixed R/W 50/50):
    ---------------------------------
    Block Size | 4k            (IOPS) | 64k           (IOPS)
      ------   | ---            ----  | ----           ----
    Read       | 117.70 MB/s  (29.4k) | 408.28 MB/s   (6.3k)
    Write      | 118.01 MB/s  (29.5k) | 410.43 MB/s   (6.4k)
    Total      | 235.72 MB/s  (58.9k) | 818.71 MB/s  (12.7k)
               |                      |
    Block Size | 512k          (IOPS) | 1m            (IOPS)
      ------   | ---            ----  | ----           ----
    Read       | 391.50 MB/s    (764) | 388.06 MB/s    (378)
    Write      | 412.30 MB/s    (805) | 413.90 MB/s    (404)
    Total      | 803.81 MB/s   (1.5k) | 801.96 MB/s    (782)
    

    Heficed (NAS, CEPH Triple Replication)

    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    #              Yet-Another-Bench-Script              #
    #                     v2020-06-20                    #
    # https://github.com/masonr/yet-another-bench-script #
    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    
    Wed 24 Jun 2020 06:09:08 PM UTC
    
    Basic System Information:
    ---------------------------------
    Processor  : Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz
    CPU cores  : 2 @ 2199.998 MHz
    AES-NI     : ✔ Enabled
    VM-x/AMD-V : ❌ Disabled
    RAM        : 1.9Gi
    Swap       : 0B
    Disk       : 20G
    
    fio Disk Speed Tests (Mixed R/W 50/50):
    ---------------------------------
    Block Size | 4k            (IOPS) | 64k           (IOPS)
      ------   | ---            ----  | ----           ----
    Read       | 33.04 MB/s    (8.2k) | 371.78 MB/s   (5.8k)
    Write      | 33.09 MB/s    (8.2k) | 373.73 MB/s   (5.8k)
    Total      | 66.13 MB/s   (16.5k) | 745.52 MB/s  (11.6k)
               |                      |
    Block Size | 512k          (IOPS) | 1m            (IOPS)
      ------   | ---            ----  | ----           ----
    Read       | 382.63 MB/s    (747) | 355.50 MB/s    (347)
    Write      | 402.96 MB/s    (787) | 379.17 MB/s    (370)
    Total      | 785.59 MB/s   (1.5k) | 734.68 MB/s    (717)
    

    OVH VPS (Local SSD)

    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    #              Yet-Another-Bench-Script              #
    #                     v2020-06-20                    #
    # https://github.com/masonr/yet-another-bench-script #
    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    
    Wed 24 Jun 2020 06:21:04 PM UTC
    
    Basic System Information:
    ---------------------------------
    Processor  : Intel Xeon E3-12xx v2 (Ivy Bridge, IBRS)
    CPU cores  : 1 @ 2599.998 MHz
    AES-NI     : ✔ Enabled
    VM-x/AMD-V : ✔ Enabled
    RAM        : 1.9Gi
    Swap       : 0B
    Disk       : 20G
    
    fio Disk Speed Tests (Mixed R/W 50/50):
    ---------------------------------
    Block Size | 4k            (IOPS) | 64k           (IOPS)
      ------   | ---            ----  | ----           ----
    Read       | 8.43 MB/s     (2.1k) | 34.39 MB/s     (537)
    Write      | 8.47 MB/s     (2.1k) | 34.77 MB/s     (543)
    Total      | 16.90 MB/s    (4.2k) | 69.17 MB/s    (1.0k)
               |                      |
    Block Size | 512k          (IOPS) | 1m            (IOPS)
      ------   | ---            ----  | ----           ----
    Read       | 55.78 MB/s     (108) | 66.54 MB/s      (64)
    Write      | 58.70 MB/s     (114) | 71.04 MB/s      (69)
    Total      | 114.48 MB/s    (222) | 137.59 MB/s    (133)
    

    Personal VM (SSD, no RAID)

    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    #              Yet-Another-Bench-Script              #
    #                     v2020-06-20                    #
    # https://github.com/masonr/yet-another-bench-script #
    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    
    Wed 24 Jun 2020 11:24:54 AM PDT
    
    Basic System Information:
    ---------------------------------
    Processor  : Common KVM processor
    CPU cores  : 4 @ 4007.996 MHz
    AES-NI     : ❌ Disabled
    VM-x/AMD-V : ❌ Disabled
    RAM        : 3.9Gi
    Swap       : 4.0Gi
    Disk       : 46G
    
    fio Disk Speed Tests (Mixed R/W 50/50):
    ---------------------------------
    Block Size | 4k            (IOPS) | 64k           (IOPS)
      ------   | ---            ----  | ----           ----
    Read       | 146.98 MB/s  (36.7k) | 154.68 MB/s   (2.4k)
    Write      | 147.37 MB/s  (36.8k) | 155.49 MB/s   (2.4k)
    Total      | 294.35 MB/s  (73.5k) | 310.18 MB/s   (4.8k)
               |                      |
    Block Size | 512k          (IOPS) | 1m            (IOPS)
      ------   | ---            ----  | ----           ----
    Read       | 208.52 MB/s    (407) | 179.76 MB/s    (175)
    Write      | 219.60 MB/s    (428) | 191.73 MB/s    (187)
    Total      | 428.12 MB/s    (835) | 371.50 MB/s    (362)
    

    Local SSDs in RAID 10 would perform better than network storage aswell, Vultr local isn't a good example as we all clearly know there's no RAID as they avoid the question. NAS/Ceph is good for high availbility but performance of local SSD/NVMe will be better (when done right) - There's still the risk of failure even in a NAS, albeit not as high.

    OVHs results look surprisingly poor though unless they're limiting?

  • @WSCallum said:

    @Brend4n said:

    @raindog308 said:

    @Brend4n said: I mean, providers that have network/redundant/CEPH storage are pretty comparable in pricing these days... UpCloud and Heficed come to mind. Heficed use CEPH with triple replication, and UpCloud their own inhouse I believe.

    What's the comparison on that vs. local SSD/NVME though?

    I think most people would prefer local RAID for performance reasons.

    Here are some tests I did with various providers. Speeds should be more than sufficient for most people, only really beat by NVMe. I'd take storage reliability any day, esp for production.

    Vultr HF (Local NVMe. No RAID)

    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    #              Yet-Another-Bench-Script              #
    #                     v2020-06-20                    #
    # https://github.com/masonr/yet-another-bench-script #
    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    
    Wed Jun 24 18:15:14 UTC 2020
    
    Basic System Information:
    ---------------------------------
    Processor  : Intel Core Processor (Skylake, IBRS)
    CPU cores  : 1 @ 3792.000 MHz
    AES-NI     : ✔ Enabled
    VM-x/AMD-V : ❌ Disabled
    RAM        : 1.9Gi
    Swap       : 0B
    Disk       : 63G
    
    fio Disk Speed Tests (Mixed R/W 50/50):
    ---------------------------------
    Block Size | 4k            (IOPS) | 64k           (IOPS)
      ------   | ---            ----  | ----           ----
    Read       | 248.47 MB/s  (62.1k) | 638.28 MB/s   (9.9k)
    Write      | 249.12 MB/s  (62.2k) | 641.64 MB/s  (10.0k)
    Total      | 497.60 MB/s (124.4k) | 1.27 GB/s    (19.9k)
               |                      |
    Block Size | 512k          (IOPS) | 1m            (IOPS)
      ------   | ---            ----  | ----           ----
    Read       | 701.29 MB/s   (1.3k) | 726.14 MB/s    (709)
    Write      | 738.55 MB/s   (1.4k) | 774.50 MB/s    (756)
    Total      | 1.43 GB/s     (2.8k) | 1.50 GB/s     (1.4k)
    

    BuyVM (Local NVMe)

    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    #              Yet-Another-Bench-Script              #
    #                     v2020-06-20                    #
    # https://github.com/masonr/yet-another-bench-script #
    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    
    Wed 24 Jun 2020 11:28:04 AM PDT
    
    Basic System Information:
    ---------------------------------
    Processor  : AMD Ryzen 9 3900X 12-Core Processor
    CPU cores  : 1 @ 3792.872 MHz
    AES-NI     : ✔ Enabled
    VM-x/AMD-V : ✔ Enabled
    RAM        : 987Mi
    Swap       : 1.0Gi
    Disk       : 270G
    
    fio Disk Speed Tests (Mixed R/W 50/50):
    ---------------------------------
    Block Size | 4k            (IOPS) | 64k           (IOPS)
      ------   | ---            ----  | ----           ----
    Read       | 292.83 MB/s  (73.2k) | 2.44 GB/s    (38.2k)
    Write      | 293.61 MB/s  (73.4k) | 2.46 GB/s    (38.4k)
    Total      | 586.45 MB/s (146.6k) | 4.91 GB/s    (76.7k)
               |                      |
    Block Size | 512k          (IOPS) | 1m            (IOPS)
      ------   | ---            ----  | ----           ----
    Read       | 2.52 GB/s     (4.9k) | 2.48 GB/s     (2.4k)
    Write      | 2.65 GB/s     (5.1k) | 2.64 GB/s     (2.5k)
    Total      | 5.18 GB/s    (10.1k) | 5.12 GB/s     (5.0k)
    

    UpCloud (NAS, MaxIOPs Inhouse)

    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    #              Yet-Another-Bench-Script              #
    #                     v2020-06-20                    #
    # https://github.com/masonr/yet-another-bench-script #
    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    
    Wed 24 Jun 2020 06:19:20 PM UTC
    
    Basic System Information:
    ---------------------------------
    Processor  : Intel(R) Xeon(R) Gold 6136 CPU @ 3.00GHz
    CPU cores  : 1 @ 2992.968 MHz
    AES-NI     : ✔ Enabled
    VM-x/AMD-V : ❌ Disabled
    RAM        : 1.9Gi
    Swap       : 0B
    Disk       : 50G
    
    fio Disk Speed Tests (Mixed R/W 50/50):
    ---------------------------------
    Block Size | 4k            (IOPS) | 64k           (IOPS)
      ------   | ---            ----  | ----           ----
    Read       | 117.70 MB/s  (29.4k) | 408.28 MB/s   (6.3k)
    Write      | 118.01 MB/s  (29.5k) | 410.43 MB/s   (6.4k)
    Total      | 235.72 MB/s  (58.9k) | 818.71 MB/s  (12.7k)
               |                      |
    Block Size | 512k          (IOPS) | 1m            (IOPS)
      ------   | ---            ----  | ----           ----
    Read       | 391.50 MB/s    (764) | 388.06 MB/s    (378)
    Write      | 412.30 MB/s    (805) | 413.90 MB/s    (404)
    Total      | 803.81 MB/s   (1.5k) | 801.96 MB/s    (782)
    

    Heficed (NAS, CEPH Triple Replication)

    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    #              Yet-Another-Bench-Script              #
    #                     v2020-06-20                    #
    # https://github.com/masonr/yet-another-bench-script #
    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    
    Wed 24 Jun 2020 06:09:08 PM UTC
    
    Basic System Information:
    ---------------------------------
    Processor  : Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz
    CPU cores  : 2 @ 2199.998 MHz
    AES-NI     : ✔ Enabled
    VM-x/AMD-V : ❌ Disabled
    RAM        : 1.9Gi
    Swap       : 0B
    Disk       : 20G
    
    fio Disk Speed Tests (Mixed R/W 50/50):
    ---------------------------------
    Block Size | 4k            (IOPS) | 64k           (IOPS)
      ------   | ---            ----  | ----           ----
    Read       | 33.04 MB/s    (8.2k) | 371.78 MB/s   (5.8k)
    Write      | 33.09 MB/s    (8.2k) | 373.73 MB/s   (5.8k)
    Total      | 66.13 MB/s   (16.5k) | 745.52 MB/s  (11.6k)
               |                      |
    Block Size | 512k          (IOPS) | 1m            (IOPS)
      ------   | ---            ----  | ----           ----
    Read       | 382.63 MB/s    (747) | 355.50 MB/s    (347)
    Write      | 402.96 MB/s    (787) | 379.17 MB/s    (370)
    Total      | 785.59 MB/s   (1.5k) | 734.68 MB/s    (717)
    

    OVH VPS (Local SSD)

    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    #              Yet-Another-Bench-Script              #
    #                     v2020-06-20                    #
    # https://github.com/masonr/yet-another-bench-script #
    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    
    Wed 24 Jun 2020 06:21:04 PM UTC
    
    Basic System Information:
    ---------------------------------
    Processor  : Intel Xeon E3-12xx v2 (Ivy Bridge, IBRS)
    CPU cores  : 1 @ 2599.998 MHz
    AES-NI     : ✔ Enabled
    VM-x/AMD-V : ✔ Enabled
    RAM        : 1.9Gi
    Swap       : 0B
    Disk       : 20G
    
    fio Disk Speed Tests (Mixed R/W 50/50):
    ---------------------------------
    Block Size | 4k            (IOPS) | 64k           (IOPS)
      ------   | ---            ----  | ----           ----
    Read       | 8.43 MB/s     (2.1k) | 34.39 MB/s     (537)
    Write      | 8.47 MB/s     (2.1k) | 34.77 MB/s     (543)
    Total      | 16.90 MB/s    (4.2k) | 69.17 MB/s    (1.0k)
               |                      |
    Block Size | 512k          (IOPS) | 1m            (IOPS)
      ------   | ---            ----  | ----           ----
    Read       | 55.78 MB/s     (108) | 66.54 MB/s      (64)
    Write      | 58.70 MB/s     (114) | 71.04 MB/s      (69)
    Total      | 114.48 MB/s    (222) | 137.59 MB/s    (133)
    

    Personal VM (SSD, no RAID)

    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    #              Yet-Another-Bench-Script              #
    #                     v2020-06-20                    #
    # https://github.com/masonr/yet-another-bench-script #
    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    
    Wed 24 Jun 2020 11:24:54 AM PDT
    
    Basic System Information:
    ---------------------------------
    Processor  : Common KVM processor
    CPU cores  : 4 @ 4007.996 MHz
    AES-NI     : ❌ Disabled
    VM-x/AMD-V : ❌ Disabled
    RAM        : 3.9Gi
    Swap       : 4.0Gi
    Disk       : 46G
    
    fio Disk Speed Tests (Mixed R/W 50/50):
    ---------------------------------
    Block Size | 4k            (IOPS) | 64k           (IOPS)
      ------   | ---            ----  | ----           ----
    Read       | 146.98 MB/s  (36.7k) | 154.68 MB/s   (2.4k)
    Write      | 147.37 MB/s  (36.8k) | 155.49 MB/s   (2.4k)
    Total      | 294.35 MB/s  (73.5k) | 310.18 MB/s   (4.8k)
               |                      |
    Block Size | 512k          (IOPS) | 1m            (IOPS)
      ------   | ---            ----  | ----           ----
    Read       | 208.52 MB/s    (407) | 179.76 MB/s    (175)
    Write      | 219.60 MB/s    (428) | 191.73 MB/s    (187)
    Total      | 428.12 MB/s    (835) | 371.50 MB/s    (362)
    

    Local SSDs in RAID 10 would perform better than network storage aswell, Vultr local isn't a good example as we all clearly know there's no RAID as they avoid the question. NAS/Ceph is good for high availbility but performance of local SSD/NVMe will be better (when done right) - There's still the risk of failure even in a NAS, albeit not as high.

    OVHs results look surprisingly poor though unless they're limiting?

    Yea, OVH is probably limiting. This VM is their black friday offer from last year. Of course, Local/NVMe/RAID will always do better "when done right". But for most people, the compromised performance wont make a difference, depending on their workload. Certainly it isn't terrible performance. What's the expected IO of RAID local non-nvme?

  • @Brend4n said:

    @WSCallum said:

    @Brend4n said:

    @raindog308 said:

    @Brend4n said: I mean, providers that have network/redundant/CEPH storage are pretty comparable in pricing these days... UpCloud and Heficed come to mind. Heficed use CEPH with triple replication, and UpCloud their own inhouse I believe.

    What's the comparison on that vs. local SSD/NVME though?

    I think most people would prefer local RAID for performance reasons.

    Here are some tests I did with various providers. Speeds should be more than sufficient for most people, only really beat by NVMe. I'd take storage reliability any day, esp for production.

    Vultr HF (Local NVMe. No RAID)

    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    #              Yet-Another-Bench-Script              #
    #                     v2020-06-20                    #
    # https://github.com/masonr/yet-another-bench-script #
    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    
    Wed Jun 24 18:15:14 UTC 2020
    
    Basic System Information:
    ---------------------------------
    Processor  : Intel Core Processor (Skylake, IBRS)
    CPU cores  : 1 @ 3792.000 MHz
    AES-NI     : ✔ Enabled
    VM-x/AMD-V : ❌ Disabled
    RAM        : 1.9Gi
    Swap       : 0B
    Disk       : 63G
    
    fio Disk Speed Tests (Mixed R/W 50/50):
    ---------------------------------
    Block Size | 4k            (IOPS) | 64k           (IOPS)
      ------   | ---            ----  | ----           ----
    Read       | 248.47 MB/s  (62.1k) | 638.28 MB/s   (9.9k)
    Write      | 249.12 MB/s  (62.2k) | 641.64 MB/s  (10.0k)
    Total      | 497.60 MB/s (124.4k) | 1.27 GB/s    (19.9k)
               |                      |
    Block Size | 512k          (IOPS) | 1m            (IOPS)
      ------   | ---            ----  | ----           ----
    Read       | 701.29 MB/s   (1.3k) | 726.14 MB/s    (709)
    Write      | 738.55 MB/s   (1.4k) | 774.50 MB/s    (756)
    Total      | 1.43 GB/s     (2.8k) | 1.50 GB/s     (1.4k)
    

    BuyVM (Local NVMe)

    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    #              Yet-Another-Bench-Script              #
    #                     v2020-06-20                    #
    # https://github.com/masonr/yet-another-bench-script #
    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    
    Wed 24 Jun 2020 11:28:04 AM PDT
    
    Basic System Information:
    ---------------------------------
    Processor  : AMD Ryzen 9 3900X 12-Core Processor
    CPU cores  : 1 @ 3792.872 MHz
    AES-NI     : ✔ Enabled
    VM-x/AMD-V : ✔ Enabled
    RAM        : 987Mi
    Swap       : 1.0Gi
    Disk       : 270G
    
    fio Disk Speed Tests (Mixed R/W 50/50):
    ---------------------------------
    Block Size | 4k            (IOPS) | 64k           (IOPS)
      ------   | ---            ----  | ----           ----
    Read       | 292.83 MB/s  (73.2k) | 2.44 GB/s    (38.2k)
    Write      | 293.61 MB/s  (73.4k) | 2.46 GB/s    (38.4k)
    Total      | 586.45 MB/s (146.6k) | 4.91 GB/s    (76.7k)
               |                      |
    Block Size | 512k          (IOPS) | 1m            (IOPS)
      ------   | ---            ----  | ----           ----
    Read       | 2.52 GB/s     (4.9k) | 2.48 GB/s     (2.4k)
    Write      | 2.65 GB/s     (5.1k) | 2.64 GB/s     (2.5k)
    Total      | 5.18 GB/s    (10.1k) | 5.12 GB/s     (5.0k)
    

    UpCloud (NAS, MaxIOPs Inhouse)

    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    #              Yet-Another-Bench-Script              #
    #                     v2020-06-20                    #
    # https://github.com/masonr/yet-another-bench-script #
    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    
    Wed 24 Jun 2020 06:19:20 PM UTC
    
    Basic System Information:
    ---------------------------------
    Processor  : Intel(R) Xeon(R) Gold 6136 CPU @ 3.00GHz
    CPU cores  : 1 @ 2992.968 MHz
    AES-NI     : ✔ Enabled
    VM-x/AMD-V : ❌ Disabled
    RAM        : 1.9Gi
    Swap       : 0B
    Disk       : 50G
    
    fio Disk Speed Tests (Mixed R/W 50/50):
    ---------------------------------
    Block Size | 4k            (IOPS) | 64k           (IOPS)
      ------   | ---            ----  | ----           ----
    Read       | 117.70 MB/s  (29.4k) | 408.28 MB/s   (6.3k)
    Write      | 118.01 MB/s  (29.5k) | 410.43 MB/s   (6.4k)
    Total      | 235.72 MB/s  (58.9k) | 818.71 MB/s  (12.7k)
               |                      |
    Block Size | 512k          (IOPS) | 1m            (IOPS)
      ------   | ---            ----  | ----           ----
    Read       | 391.50 MB/s    (764) | 388.06 MB/s    (378)
    Write      | 412.30 MB/s    (805) | 413.90 MB/s    (404)
    Total      | 803.81 MB/s   (1.5k) | 801.96 MB/s    (782)
    

    Heficed (NAS, CEPH Triple Replication)

    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    #              Yet-Another-Bench-Script              #
    #                     v2020-06-20                    #
    # https://github.com/masonr/yet-another-bench-script #
    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    
    Wed 24 Jun 2020 06:09:08 PM UTC
    
    Basic System Information:
    ---------------------------------
    Processor  : Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz
    CPU cores  : 2 @ 2199.998 MHz
    AES-NI     : ✔ Enabled
    VM-x/AMD-V : ❌ Disabled
    RAM        : 1.9Gi
    Swap       : 0B
    Disk       : 20G
    
    fio Disk Speed Tests (Mixed R/W 50/50):
    ---------------------------------
    Block Size | 4k            (IOPS) | 64k           (IOPS)
      ------   | ---            ----  | ----           ----
    Read       | 33.04 MB/s    (8.2k) | 371.78 MB/s   (5.8k)
    Write      | 33.09 MB/s    (8.2k) | 373.73 MB/s   (5.8k)
    Total      | 66.13 MB/s   (16.5k) | 745.52 MB/s  (11.6k)
               |                      |
    Block Size | 512k          (IOPS) | 1m            (IOPS)
      ------   | ---            ----  | ----           ----
    Read       | 382.63 MB/s    (747) | 355.50 MB/s    (347)
    Write      | 402.96 MB/s    (787) | 379.17 MB/s    (370)
    Total      | 785.59 MB/s   (1.5k) | 734.68 MB/s    (717)
    

    OVH VPS (Local SSD)

    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    #              Yet-Another-Bench-Script              #
    #                     v2020-06-20                    #
    # https://github.com/masonr/yet-another-bench-script #
    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    
    Wed 24 Jun 2020 06:21:04 PM UTC
    
    Basic System Information:
    ---------------------------------
    Processor  : Intel Xeon E3-12xx v2 (Ivy Bridge, IBRS)
    CPU cores  : 1 @ 2599.998 MHz
    AES-NI     : ✔ Enabled
    VM-x/AMD-V : ✔ Enabled
    RAM        : 1.9Gi
    Swap       : 0B
    Disk       : 20G
    
    fio Disk Speed Tests (Mixed R/W 50/50):
    ---------------------------------
    Block Size | 4k            (IOPS) | 64k           (IOPS)
      ------   | ---            ----  | ----           ----
    Read       | 8.43 MB/s     (2.1k) | 34.39 MB/s     (537)
    Write      | 8.47 MB/s     (2.1k) | 34.77 MB/s     (543)
    Total      | 16.90 MB/s    (4.2k) | 69.17 MB/s    (1.0k)
               |                      |
    Block Size | 512k          (IOPS) | 1m            (IOPS)
      ------   | ---            ----  | ----           ----
    Read       | 55.78 MB/s     (108) | 66.54 MB/s      (64)
    Write      | 58.70 MB/s     (114) | 71.04 MB/s      (69)
    Total      | 114.48 MB/s    (222) | 137.59 MB/s    (133)
    

    Personal VM (SSD, no RAID)

    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    #              Yet-Another-Bench-Script              #
    #                     v2020-06-20                    #
    # https://github.com/masonr/yet-another-bench-script #
    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    
    Wed 24 Jun 2020 11:24:54 AM PDT
    
    Basic System Information:
    ---------------------------------
    Processor  : Common KVM processor
    CPU cores  : 4 @ 4007.996 MHz
    AES-NI     : ❌ Disabled
    VM-x/AMD-V : ❌ Disabled
    RAM        : 3.9Gi
    Swap       : 4.0Gi
    Disk       : 46G
    
    fio Disk Speed Tests (Mixed R/W 50/50):
    ---------------------------------
    Block Size | 4k            (IOPS) | 64k           (IOPS)
      ------   | ---            ----  | ----           ----
    Read       | 146.98 MB/s  (36.7k) | 154.68 MB/s   (2.4k)
    Write      | 147.37 MB/s  (36.8k) | 155.49 MB/s   (2.4k)
    Total      | 294.35 MB/s  (73.5k) | 310.18 MB/s   (4.8k)
               |                      |
    Block Size | 512k          (IOPS) | 1m            (IOPS)
      ------   | ---            ----  | ----           ----
    Read       | 208.52 MB/s    (407) | 179.76 MB/s    (175)
    Write      | 219.60 MB/s    (428) | 191.73 MB/s    (187)
    Total      | 428.12 MB/s    (835) | 371.50 MB/s    (362)
    

    Local SSDs in RAID 10 would perform better than network storage aswell, Vultr local isn't a good example as we all clearly know there's no RAID as they avoid the question. NAS/Ceph is good for high availbility but performance of local SSD/NVMe will be better (when done right) - There's still the risk of failure even in a NAS, albeit not as high.

    OVHs results look surprisingly poor though unless they're limiting?

    Yea, OVH is probably limiting. This VM is their black friday offer from last year. Of course, Local/NVMe/RAID will always do better "when done right". But for most people, the compromised performance wont make a difference, depending on their workload. Certainly it isn't terrible performance. What's the expected IO of RAID local non-nvme?

     # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    #              Yet-Another-Bench-Script              #
    #                     v2020-02-10                    #
    # https://github.com/masonr/yet-another-bench-script #
    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    
    Mon 15 Jun 2020 03:19:45 PM BST
    
    Basic System Information:
    ---------------------------------
    Processor  : AMD EPYC 7302P 16-Core Processor
    CPU cores  : 32 @ 3286.780 MHz
    AES-NI     : ✔ Enabled
    VM-x/AMD-V : ✔ Enabled
    RAM        : 188Gi
    
    fio Disk Speed Tests (Mixed R/W 50/50):
    ---------------------------------
    Block Size | 4kb           (IOPS) | 64kb          (IOPS)
      ------   | ---            ----  | ----           ----
    Read       | 203.04 MB/s  (50.7k) | 1.23 GB/s    (19.2k)
    Write      | 203.57 MB/s  (50.8k) | 1.23 GB/s    (19.3k)
    Total      | 406.62 MB/s (101.6k) | 2.47 GB/s    (38.6k)
               |                      |
    Block Size | 512kb         (IOPS) | 1mb           (IOPS)
      ------   | -----          ----  | ---            ----
    Read       | 1.03 GB/s     (2.0k) | 1.33 GB/s     (1.3k)
    Write      | 1.09 GB/s     (2.1k) | 1.42 GB/s     (1.3k)
    Total      | 2.13 GB/s     (4.1k) | 2.75 GB/s     (2.6k)
    
    iperf3 Network Speed Tests (IPv4):
    ---------------------------------
    Provider                  | Location (Link)           | Send Speed      | Recv Speed
                              |                           |                 |
    Bouygues Telecom          | Paris, FR (10G)           | 933 Mbits/sec   | 935 Mbits/sec
    Online.net                | Paris, FR (10G)           | busy            | busy
    WorldStream               | The Netherlands (10G)     | 938 Mbits/sec   | 938 Mbits/sec
    wilhelm.tel               | Hamburg, DE (10G)         | 931 Mbits/sec   | 895 Mbits/sec
    Biznet                    | Bogor, Indonesia (1G)     | 601 Mbits/sec   | 84.5 Mbits/sec
    Hostkey                   | Moscow, RU (1G)           | 916 Mbits/sec   | 920 Mbits/sec
    Velocity Online           | Tallahassee, FL, US (10G) | 154 Mbits/sec   | 287 Mbits/sec
    Airstream Communications  | Eau Claire, WI, US (10G)  | 848 Mbits/sec   | 369 Mbits/sec
    Hurricane Electric        | Fremont, CA, US (10G)     | busy            | busy
    
    Geekbench 4 Benchmark Test:
    ---------------------------------
    Test            | Value
                    |
    Single Core     | 3859
    Multi Core      | 48175
    Full Test       | https://browser.geekbench.com/v4/cpu/15565453

    That's one I did the other week with 4 x SSDs RAID 10, as an example.

  • I was just considering using Vultr's cloud offerings for a database instance -- specifically the high frequency VPS with NVMe storage.

    But now I'm really having doubts, and the more I look for info on their infrastructure, the more concerned I get.

    For me, one of the big reasons to use cloud services instead of dedicated hardware is because the provider handles hardware issues behind the scenes, ensuring high levels of reliability and availability without the customer needing to do anything or even know about the details. That should mean that they're using high quality hardware with built-in redundancies. Otherwise, I'd feel better managing my own server, where at least I can set up some basic software RAID-1 and know that a bad SSD/NVMe won't kill my database.

  • Brend4nBrend4n Member
    edited June 2020

    @aj_potc said:
    I was just considering using Vultr's cloud offerings for a database instance -- specifically the high frequency VPS with NVMe storage.

    But now I'm really having doubts, and the more I look for info on their infrastructure, the more concerned I get.

    For me, one of the big reasons to use cloud services instead of dedicated hardware is because the provider handles hardware issues behind the scenes, ensuring high levels of reliability and availability without the customer needing to do anything or even know about the details. That should mean that they're using high quality hardware with built-in redundancies. Otherwise, I'd feel better managing my own server, where at least I can set up some basic software RAID-1 and know that a bad SSD/NVMe won't kill my database.

    Yup, likewise. I have a colo with ~20 i7 servers; however, host prod projects on cloud providers with redundant/NAS storage. Better safe than sorry, esp. if I dont need much compute resources.

    Check out UpCloud, you'll get comparable MySQL, Redis, IO performance with redundant/NAS storage.
    https://joshtronic.com/2020/06/01/vps-showdown-digitalocean-lightsail-linode-upcloud-vultr/

  • @raindog308 said: What's the comparison on that vs. local SSD/NVME though?

    I think most people would prefer local RAID for performance reasons.

    For workload that require very low latency local SSD/NVMe would be the best choice. There is overhead network latency when you use Ceph.

    You also only shared those local SSD with other VPS in same node while with Ceph you probably must compete in a single cluster that shared with much higher, even thousand VPS. That's why some provider should limit their IOPS quota for each VPS, unless you want small user abusing the cluster.

    The bright side Ceph aside for high reliability is they can be scale very well. If you need additional IOPS you just need add new node. And you can get block storage, object storage (S3), and file system from single cluster which is simplified operational.

    Hetzner with Ceph

    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    #              Yet-Another-Bench-Script              #
    #                     v2020-06-20                    #
    # https://github.com/masonr/yet-another-bench-script #
    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    
    Thu Jun 25 03:49:52 CEST 2020
    
    Basic System Information:
    ---------------------------------
    Processor  : Intel Xeon Processor (Skylake, IBRS)
    CPU cores  : 2 @ 2099.998 MHz
    AES-NI     : ✔ Enabled
    VM-x/AMD-V : ❌ Disabled
    RAM        : 3.8G
    Swap       : 0B
    Disk       : 38G
    
    fio Disk Speed Tests (Mixed R/W 50/50):
    ---------------------------------
    Block Size | 4k            (IOPS) | 64k           (IOPS)
      ------   | ---            ----  | ----           ---- 
    Read       | 9.26 MB/s     (2.3k) | 160.40 MB/s   (2.5k)
    Write      | 9.29 MB/s     (2.3k) | 161.24 MB/s   (2.5k)
    Total      | 18.55 MB/s    (4.6k) | 321.64 MB/s   (5.0k)
               |                      |                     
    Block Size | 512k          (IOPS) | 1m            (IOPS)
      ------   | ---            ----  | ----           ---- 
    Read       | 523.14 MB/s   (1.0k) | 658.95 MB/s    (643)
    Write      | 550.94 MB/s   (1.0k) | 702.83 MB/s    (686)
    Total      | 1.07 GB/s     (2.0k) | 1.36 GB/s     (1.3k)
    

    Hetzner probably limit IOPS to 5K for each VPS.

  • nbnnbn Member
    edited June 2020

    @Brend4n said:

    @nbn said:
    I've had great luck with the high frequency VPS from vultr, and they aren't much of an upcharge. You get a beast ryzen cpu and I haven't had any crashes. Shit happens, but vultr now also has snapshots making backups really easy to create...

    Vultr uses Ryzen now? :open_mouth:

    Oh they actually might be high GHZ intel skylake, my bad. They were hellishly fast so I assumed ryzen

  • jlayjlay Member
    edited June 2020

    @raindog308 said:

    @Brend4n said: I mean, providers that have network/redundant/CEPH storage are pretty comparable in pricing these days... UpCloud and Heficed come to mind. Heficed use CEPH with triple replication, and UpCloud their own inhouse I believe.

    What's the comparison on that vs. local SSD/NVME though?

    I think most people would prefer local RAID for performance reasons.

    The difference can be less drastic than you may think

    In our work we've done a lot with RDMA storage clustering, works worlds better than TCP. The network effect gets compounded by latency, which this cuts by an order of magnitude

    Different solutions have different problems - I know GlusterFS the best. The per file metadata can hurt random iops the most - but host level disk caches solve this in VMs

Sign In or Register to comment.