All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Looking for the most cost effective SSDs
randvegeta
Member, Host Rep
We are considering replacing our HDDs in our cluster with SSDs.
We are running Virtuozzo Storage with DC grade HDDs of various brands (Seagate, WD and HGST). Storage cluster runs over a 10G network and we use SSDs for read/write caching to boost performance.
SSDs have traditionally been too expensive to use in the cluster, especially given data is triplicated (so we can only use 1/3 of the actual disk space). But prices have really started to come down; even more so for consumer grade SSDs.
When comparing Enterprise SSDs to HDDs, the price is easily 20x higher for SSDs per GB. But the cheapest consumer grade SSDs are only around 6x the price per GB.
A new Crucial MX300 2TB SSD can be purchased for around US$500, while a WD Gold 4TB HDD is about $160. Given the data is stored in triplicate across the cluster, I figure the reliability is of SSD are less of a concern. I am wondering however if there would be any significant performance boost from using these consumer grade SSDs, and if the benefits would justify the added cost.
One must consider the performance of pure SSD vs (Enterprise) HDD with SSD caching, cost and lifespan. Some HDDs have very long life-spans, but 'cheap' SSDs have a reputation for dieing young. If the SSD cost 6x as much per GB but lasts only half as long, then it actually costs 12x as much.
Anyone on LET running an SSD storage cluster? Anyone with experience with the MX300 series SSDs?
Comments
Yea... no, not all SSDs are equal and the price is not as much factor as it seems.
http://techreport.com/review/27909/the-ssd-endurance-experiment-theyre-all-dead
MX300 are ok, i have a 275Gb M.2 as USB-C stick (3.1, 10Gbit) which does work fine.
BX200 are notably horrible, only SSDs we ever sent back to vendor (was an Amazon deal, even for that price not worth it).
OCZ miserable, Corsair good, Samsung good, Kingston good - Intel last used as 3Gbit ones, not good, later probably better though.
Sandisk surprisingly good overall - ULTRA II but also the SLC cached SSD PLUS (which is somewhat like a BX200, just working far better).
Enterprise SSDs offer a few advantages but are not worth the price increase:
AHCI PCIe SSDs solve 1 (and i think 2) and NVMe SSDs solve 1 and 2 plus more IOPS in certain scenarios.
To the contrary, many reviews have shown that MLC and TLC SSD's endurance exceeds their marketed specification. (http://techreport.com/review/27909/the-ssd-endurance-experiment-theyre-all-dead). The MX300 (and all SSD's) have a published write endurance, look at your data write trends and see if they are within that specification. I have personally experienced under 0.5% SSD failures year over year, while I've experienced >2% spinning drive failures year over year (at enough scale to be statistically meaningful).
You'll likely start to see bus saturation way before you see storage saturation if you are running enough disks. If you need more IOPS, SSD's are the way to go. Tiered storage is probably still the most cost effective means of storage. I would also look at NVME based storage, which can easily do 250k-400k iops.
Actually, caching really helps with performance, particularly with containerized (VZ) VPS. But I'm seeing relatively poor read/write performance on VMs. At the moment, our cluster is underutilized, but the performance for VMs just isn't that great. With all the extra disks and caching, performance is expected to be much better... And this is why we are thinking about switching to SSDs.
What exactly do you mean by Tiered storage?
...which MX300 (and all SSD's) tend to overshoot by an insane margin.
The 275 GB model is spec'ed for 80 TB written total.
It has endured 2275 TB and is still going: https://3dnews.ru/938764/page-2.html#Crucial MX300
Of course there are also a number of failed drives in that test (some of which did not even survive through their promised lifespan).
Basically, don't even bother talking about manufacturers in most cases. The only useful piece of info this list gives, is about Samsung. That's the only company which uses their own in-house controllers and in-house flash. The rest use a hodgepodge of OEM sourced controllers, flash, or even entire pre-assembled and ready to go boards. Which is NOT a bad thing, you just have to keep in mind to look deeper than the name brand on the box. You need to know Marvell, SandForce, Phison (and MLC, or 2D/3D TLC and at which process) -- not simply Kingston, OCZ or Corsair.
Old but gold. Read that article a while back. But the price was still too high to even consider. The MX300 is the only drive at the moment I consider to be worth switching to (given the cost). At least it's the only one I know of with that size/price ratio. Other drives just seem too expensive (or too poor). Even the MX300 is only worth considering if it can handle the read/write cycles for a comparable time to the 4TB HDDs we've currently got in deployment.
@randvegeta what do you use for caching both in terms of software and hardware ? As This might be a problem.
I've had good experiences with Samsung, and the reviews have been almost all positive. But the 850 EVO is a good $100 more expensive than the MX300 for the 2TB model, and that's hard to swallow. For my cluster, that makes each 1TB cost >US$1,000.
Samsung 850 EVO 2TB SSD = $635 (ish)
Real disk capacity is only 1.8TB (if that)
Cost per TB = US$352
Cost per TB in the cloud = US$1,056
Pretty pricey for storage. Using WD Gold HDDs, the cost (in the cloud) per TB is just US$135 /TB. The >$1k price tag is a massive jump.
Using Virtuozzo Storage, so caching is done by the software. We are using Kingston DC400 480GB SSDs. For caching, we can only use 'Enterprise' SSDs since we need the power-loss-protection functionality. No consumer grade SSD that I am aware of has such a feature.
Ok, no experience with that so I can't help you much.
We use Intel CAS on Intel DC and Samsung PM863 drives and it works quite well in front of SAS 10k, of course nowhere close to raw SSD performance.
Do you store data in triplicate across multiple nodes over 10G network?
If so, what kind of read/write speeds do you get?
Not on this particular solution. That's a solution we build for a Client with local storage only. It's designed for cold storage, but with some information being frequently required which i admit fits ideally into the use case of Intel CAS. I can't say much about real world VM performance on this solution unfortunately.
Does the bench.sh script provide any real/meaningful data on disk performance?
On the same storage cluster, I'm getting crazy difference in performance depending on weather or not I'm using a container or a VM.
With a container, I'm getting near 1Gbyte/s I/O speeds, whereas I don't even get 100Mbyte/s on the VM. Same storage cluster.. same hardware...
Ploop?
Not yet today, but thanks for asking.
Samsung 845DC EVO looks interesting. At $230/500MB it's a bit more expensive than the MX300 but it has power fail capacitors, and more overprovisioning for more write wear capacity. I'd consider.
Not familiar with this model, and does not seem to be available in HK at all.
Hmm.. Well it seems looking to replace all the HDDs may be completely unnecessary. The guys at Virtuozzo have stated that there shouldn't be such a disparity between CT and VM disk performance, and so there must be some issue with the configuration or software.
My cluster i still running on version 6 but I setup a version 7 test cluster using sub-optimal hardware (Gbit network, no SSD cache, and on relatively low end CPUs) and the disk I/O performance was superior to my live system. Granted you cannot really compare an active system to an unused and empty one, but still....
Long story short, upgrading the software to version 7 may provide the boost I'm looking for, while upgrading to SSDs may provide little or no benefit at all, but at much higher cost.
Corsair uses custom controllers based on Marvell. Samsung does NOT always use Samsung flash. Kingston PCIe drives are also custom Phison.
Nobody aside of cheap Chinese brands use stock controllers anymore at this time.
You also entirely forgot to mention Intel which uses a various flash manufacturers and their own controllers.
Regardless of this, Sandforce based SSDs despite using the same controller overall and only 3 different flash vendors had INSANE differences in durability with OCZ coming out WAY far on the bottom.
You ideally buw SSDs in Taiwan, it has no/low import tax to HK and far better pricing aside of Samsung.
Is the pricing really better in Taiwan?
Can you recommend any online stores from which they can be ordered?
Hello,
I state the fact the SSD you could get have better performances than your actual HD, unless you need to improve performances i would not do that.
Are you planing to change all the cluster storage one day ?
I need to improve the performance of our VMs because right now we are getting performance similar to what you see in a RAID 1 setup, with sub 100MB/s write speed. According to Virtuozzo's own white sheets, the performance should be comparable to local disk (no-raid) which would be around 150-200MB/s. And with SSD caching (which we have) 300MB/s +
SSD cache is supposed to give us near SSD performance without actually having all SSDs. Which of course would be ideal.
Well it's not realistic to swap out 10s of TB of disks in 1 day, but it would have happened over the course of a week or so. But now we are looking at upgrading the software instead, which similarly would be done over a week. Upgrading nodes 1 by 1, for however long it takes to complete without downtime to the end user.
Ok if i were you i would make an estimation of the costs, change only the disks / change the all storage system + time required in the two scenarios.
look at PM863A if you're looking at Samsung and quality.
Not all SSDs are the same when it comes to performance...
https://b3n.org/ssd-zfs-zil-slog-benchmarks-intel-dc-s3700-intel-dc-s3500-seagate-600-pro-crucial-mx100-comparison/
Well I was basing this on MX300 2TB prices. We have about 60TB of raw space in our cluster, which would mean 30x SSDs at a cost of around US$15,000.
If this boosts read/write speed to 400MB/s + then it may be worth it, but as mentioned above, it may not be at all neccessary.
We do caching now, using much more costly 'Enterprise' grade SSDs (Kingston). With containers, we get incredible speeds, but the VMs are running at 1/10 the speed. The VZ guys have assured us that this is not normal, and if we can boost performance upto 250MB/s + then that would be acceptable for us... And apparently save us 15k!
These look nice! I'll see if I can source some in HK (or TW as @William suggested). You use these in production now?
I'll probably add another Pure SSD storage tier for very high performance stuff. Won't need such a huge capacity to start with so the cost will be much more manageable.
I don't have all the clues but i'd say you might need to do some setup. Some systems have fixed values regarding storage. That could be the reason you dont have the expected performance but maybe we should start another discussion in "debug mode".
We've deployed thousands of SM863, PM863, and PM1633 models - they're superior both in performance but also in reliability in environments getting tons of data read/written every single day (CDN edge servers).
I can really recommend them.