New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
@jeanluc @EtienneM - Few comments to making it clear:
1) My comments are only about OVH VPS Cloud servers (not public cloud, it's completely separate product / service)
2) Generic benchmark is quite useless. I've got bunch of servers, and the performance variance is staggering. Some servers fly, and others are extremely slow. But this only applies mostly to the CEPH backed storage system. And the situation can change dynamically. So even if you get fast server, it could be slow after a month and so on. The classic noisy neighbors problem.
3) OVH Guarantees only 200 IOPS. Even that would be fine for most of use cases. The actually problems begin, when single IOPS starts taking seconds or even tens of seconds.
4) Our servers in SBG have been working pretty well. A few servers in GRA have had more serious issues.
5) There's one problem related to the Windows OS. I really don't know if it's Windows, some drivers, VM environment or what. But when the disk I/O gets slow enough. It dies completely and won't resume without hard reboot. I've been trying to ask more details about this from several forums, but nobody seems to know anything about this problem. Yet, we've been seeing this problem with OVH and other providers (when their storage systems fail). - If anyone knows anything about this, please contact me. - This clearly isn't OVH specific issue.
6) Because we've still got VPS Cloud servers at OVH. The situation is quite good. Maybe one crashed system per month or so. For the performance in general, you'll get what you pay for. - Only thing which made us mad, was the related hangs from point 5.
7) Of course you can get better performance from their Public Cloud series, or from UpCloud (Which is awesome), and of course more expensive, but provides great bang for buck. and so on. But for our low end customers, the OVH VPS Cloud is perfect fit, if the dips just wouldn't be just as deep as those were.
Hi,
Did not had time to check your messages sorry. I check tomorrow
@jeanluc
Hi,
I asked the team, they are thinking about it
@webDude
Can you please pm me the name of your vps that has issue, I'll check something
as of 2017 Nov.
It is pretty solid and reliable iops on all.
but there are rules that you must know.
for your information
they will offer 600 iops raid storage on dedicated servers if you un-raid it and make it double storage then you get 300 iops. so if you are streaming or going to do wordpress on it you better upgrade it to ssd and it will be over 10k iops all the time.
you know this is physically right. your desktop is 300 iops.
for public cloud,
it is 2000 iops, you can also upgrade it to ssd and will get you up to 4000 iops
you can also add storage on it, it will be cheap stealing price to "classic" or a little bit more cost to "high performance"
cheap one is like 200iops for storage purpose. so don't even try to host something to your customers.
high performance you will get up to 3000 iops.
for vps,
you will get 300 iops max, and additional disks are all 2000 iops.
i understand this is "lowend" talking forum. you will basically cry if you miscalculate the cost for your business. you gotta be smart on this part.
ovh prices are very reasonable and efing cheap compared to others.
All you're saying is very true. But as you might have noticed here, I wasn't complaining about 'general performance'. But the random peaks where the performance gets ultra low or causes systems to crash completely.
We don't require high performance systems, and that's not what I've been even asking for. Providers like UpCloud provide 100% solid 100k IOPS all the time. And it's only slightly more expensive. So immediately if we know that the constant IO requirement is high, we'll move servers to another platform.
Btw. Servers at SBG seem to be working generally just a bit faster after the big "SBG outage".
Also at GRA there has been strange system crashes exactly at midnight (CET). But that's being investigated.
If I wouldn't be happy about the servers in general, I wouln't have such a quantity of servers at OVH as we currently do. So generic quality is good, only thing I were complaining the quality and performance exceptions.
Also unfortunately many customers are very price critical. It's very easy to say that you can fix this issue by starting to use Azure, GCP or AWS. But that's going to shift the services to another price category too.
For the tasks / services which can't be handled properly with the "VPS Cloud 3", I've moved to UpCloud. - Usually at that level, it's also for the customers to pay more.
All of our systems are also working in fully replicating and offline modes. So all clients do have local copy of the database. In case the server goes down. It won't even stop the clients from doing their job. - That's something which is highly recommended design for any application. Cloud servers and networking is always "unreliable", for multiple reasons.
Btw. I've got a few $25 free credit invites for UpCloud if anyone is interested.
I've always believed that if you don't want noisy neighbors, get a dedi.
Having dedi without clusterization and distributed backend system and load balancing front, immediately means that you've got unreliable system. You're just hanging by the thread and trusting one server. That's why dedi is a bad solution, compared to much more reliable clusterized and shared system.
Having shared system wouldn't mean that you would suffer from neighbours, nor it means that performance would be bad.
ceph? I'm amazed. What a joke. Frankly, this whole horror story just confirms that ceph is working as is to be expected (assuming the brain is activated).
I'm surprised that a company like ovh (whom until now I took to be utterly bureaucratic and somewhat anal but generally a professional operation) actually does funny CFC experiments (CFC = clusterfuck crap).
About the only positive thing I can say about ceph CFC is that it recommends xfs whis is, indeed, a fine file system.
No it isn't. You're affected by hardware, software, power, storage system a networking issues. Sure, things could work for 5 years without a hitch, but one day the problems do hit your system. - I've been doing this stuff 20 years, and ... Seen that happening over and over again.
This is exactly why there's requirement for solutions like Spanner.
They? OVH? They offer the iops you can get from the drives - it also depends on the raid you choose, if your iops are sequencial or not - by no means you'll get 300 eller 600 random IO from spinning rust.
Same goes for SSD, you can max out an SSD with less than 10k iops if you "do it right" - there's no guarantee you'll be over 10k all the time for real production workloads.
So which part of Public Cloud gives 4k iops?
I have a special VPS then, I get more than 300 iops :-)
Not sure what "CET" you live in - they went offline in GRA at around 02.15 (Europe/Paris timezone), and connectivity was gone for 1-2 minutes.
Same affects you in the "CLOUD", the only difference is, that there are failover mechanisms in place.
The issue is, its getting more complex and vulnerable to things which a dedi would not get affected.
Additional there comes also stuff, like neighbours, which you tried to avoid by using a dedi, compared to my VM's I rented somewhere, my dedis have far less issues in average.
A drive or motherboard fails every 1-2 Years, but you got backups right?
Go and setup a simple failover, nothing to complex like CEPH, problem fixed.
if you say so, their 'default raid 1' for dedicated servers will be 300 iops which is the minimum raid option.
i don't know if you ever had ssd dedicated server there, ovh dedicated servers are all 'raid' drives not just 'single' drive. and i don't know what your 'do it right' meaning, i'm talking about the fio results.
for public cloud ssd, they are now 6000 iops not 4000 iops from my old instances, that's even faster.
if you thought it was slower, that will be 'sandbox' public cloud shared instances which is just 2000 iops.
recently they upgraded all hardwares and they have only SSD option only.
those 2000iops (3x HABLOCK) 4000iops (OLD SSD) form my previous post, that's the before they upgraded hardwares like a month ago.
for vps, i was talking about the vps with ceph. not vps with ssd (heard they do 1000iops).
honestly i do not know about this part because i never deal with vps, i just needed one little windows 2012 server for my off-project since vps ceph the only option for me to do windows server.
i have many servers in public cloud and dedicated too.
they were all steady and never gets "noise neighbor" effects during 18 months until now.
i only had bandwidth issue with them for about few hours in just 2 days. that's only one time, they had to do it because of public cloud hardware upgrades that i mentioned above.
Ohh. I get what your saying now, I really dont use their VPSes. they have VPS SSD, VPS CLOUD and Public Cloud. I thought you were talking about VPS in 'Public Cloud'. (Public Cloud VPS doens't exist anymore)
I never deal with them, I only have 1 'vps cloud' because of windows project and I also found out my VPS CLOUD has very low IOPS. and yes mines Ceph / NVMe I guess we all should avoid their VPS CLOUD lines. I think it's badly shared.
however, in early 2016, I had VPS SSD, it worked fine for few months without issues then I moved on to Public Cloud and Dedicated servers.
However,
Public Clouds do the separate operation than OVH VPS. with almost same price, you will get 'reserved' hardware allocation. you should try this if you have not. trust me, you will never face that issue you have. it is like having dedicated server with guaranteed resources. I have alot of servers in Public Clouds projects. serving about 3000 IPs.
No - you clearly do not know how hardware works:
Which fio options you use? Some biased unrealistic options, or something that fits your work load?
The thing is, I base my numbers on real production workload, which can be very hard to replicate with fio, because you will have a mix of sequential and random IO, the size of the data you write can differ, thus influence how well the drive (both SSD and spinning disks) will handle that particular workload, it depends on your disks current workload and queue depth.
fio is great, but it often renders useless results anyway. You can use it as a measure to see if your disk performance became worse over time if you have metrics, but that's really about it - oh and then generating some weird data that isn't useful.
I took the time to spin up both sandbox and general purpose instances across all regions - numbers differs on certain locations, but respawning the instance might render the number different.
I also happen to have a sandbox public cloud shared instance doing 11k iops, some doing 4k and some doing 500 iops - even within same region.
Also it didn't upgrade a month ago - the change has been there longer, but anyway - time doesn't matter.
Depends on what hypervisor you end on - generally speaking, it's correct you won't see much "steal" even on the sandbox machines, iops can differ a lot, both for the sandbox and every other thing - all new regions are with local raid - so if all on the same hypervisor happen to do IO that isn't optimal for the drives, then you can experience "noise" or sub-optimal performance.
It really depends on what people do as well - you might not even notice if there's a problem with performance related stuff, if you happen to not hit those specific limits on a specific box - we all have different workloads (except those who love idling machines), if a certain problem will affect us or not, really depends on the problem - that's also why there's so many different hosting providers - because they all can deliver different services to satisfy the needs of different customers.
you can run this 4k test on all of your server. will get the same result even at peak time.
i'm not sure on shared sandbox environments, but others are all the same result, slightly different at times.
fio --name=rand-write --ioengine=libaio --iodepth=32 --rw=randwrite --invalidate=1 --bsrange=4k:4k,4k:4k --size=512m --runtime=120 --time_based --do_verify=1 --direct=1 --group_reporting --numjobs=1
p.s. --numjobs=x is number of cores depends on servers.
sandboxes are raid 10 so i guess they do it with 4 ssds.
other public clouds, they don't mention it but the result is steady whole time.
how many providers do SSD raids nowadays that is fixed resource allocation unless having dedicated server plan?
at least i get the 'dedicated' server result.
i know how hardware works and iops. my hardest production work loads in real world was serving full '1Gbps dedicated line maxed out for video streaming. regardless how fast raid setup is, wihtout SSD that's impossible to make it happen with effiency cost.
well i just jumped into this topic because i'm having good experience with OVH.
some people need to know they've got some unique system without ripping off people like most of start up companies.
As I expected - the test you use is completely useless:
Raid 10 might be faster - in this case no.
Plenty of providers.
If 4k iops fio test above is "dedicated" - then your dedicated servers must suck.
"I know how hardware works and iops", yet your comments clearly show, that you don't.
Your hardest workload being 1Gbps dedicated line for video streaming:
Now in your specific case, video streaming generally speaking is very easy to deliver in terms of disk performance and iops, why? Because you're serving "large" files (let's assume 5-10 megabyte or more) - that makes it super easy for the disks to actually handle this - even for spinning disks.
You won't have a lot of random IO for video streaming, and most active data anyway will be in the file system cache.
I know exactly that, because I happened to deal with video streaming myself doing 15-18 gigabit per box, and iops really wasn't an issue.
also I don't think they are local raids, only Sandboxes are Local Raid 10 SSD.> @Zerpy said:
you typically have no experiences at all.
we are not transfering a file to a single user. that spindle HGST Ultrastar can't do this at all.
there are many users accessing different files and video streaming is all about IOPS.
please dont lie without experiences. If anyone who do the streaming server will laugh hard at you.
how do you fio then?
EDIT: 18 gigabit per box? lol where did you get that box from? and how many concurrent connection?
EDIT2: "You rarely run with a consistent 4k block size range
You rarely run with a constant 32 iodepth
You rarely do only random writes in your environment
The size of your test file is too small" --->> i loled again.
B2-*, C2-*, R2-*, G2-*, S1-* - they all use local raid, it even says on their websites.
It was also announced back in 2016.
Only older generation of the flavors (also known as EG, HG, SP) had either ceph based HA storage or non-RAID SSD.
I don't do fio tests, because they're inaccurate and do not reflect anything realistic - but at least make the size of the file, bigger than your RAM, make the block size something like --bsrange=4k:64k,4k:64k and do random rw, rw, randread, read, randwrite, write, change the iodepth - and run another 1000 tests to get results from different kinds of io load tests.
Doesn't matter if it's a single user or 1000 users (good luck with a decent quality on a gigabit line with 1k users)
Since you're doing video streaming, it means you'll either have very very hot content (this means that majority or the full hot content will stay in memory, so disk reads will be limited anyway).
If you're having hot content with a long tail, iops will be worse - but we're talking about 1 gigabit, so the actual amount of different videos you can push on a single gigabit is going to be minimal regardless.
And spindles HGST Ultrastar can actually do it - we're talking about mostly sequential reads and reading very large chunks of data (because videos tend to be more than a megabyte).
Even if you do HLS which is purely chunks of 0.5 to 2-3 megabyte, spinning disks would be perfectly fine.
In case it's not possible to do it - then just paste 30-40 minutes of "iostat -x -m 1" on LET so we can see that your iops and sequential reads exceeds what a HGST Ultrastar or two can pull off.
It's video streaming mate :-) Not rocket science.
You mean experience as in, working on a CDN focusing on video streaming and doing 600Gbps+ traffic?
Your biggest workload is 1 gigabit, my biggest is 600Gbps+ :-) So I must have 600x more experience than you or?
I don't do fio tests, because they're inaccurate and do not reflect anything realistic - but at least make the size of the file, bigger than your RAM, make the block size something like --bsrange=4k:64k,4k:64k and do random rw, rw, randread, read, randwrite, write, change the iodepth - and run another 1000 tests to get results from different kinds of io load tests.
I don't need 1000 tests, 2 minutes are enough and I would do rather run it at every hour to check I would still get the same iops and don't get noise neighbor effect.
Doesn't matter if it's a single user or 1000 users (good luck with a decent quality on a gigabit line with 1k users)
Since you're doing video streaming, it means you'll either have very very hot content (this means that majority or the full hot content will stay in memory, so disk reads will be limited anyway).>
Stay in memory? what did you stream? 240p videos? that would fit in memory?
I kind of understand that your knowledge is based on very outdated documents.
and I don't do video streaming, that was part of my in real-world experience that you never had.
If you're having hot content with a long tail, iops will be worse - but we're talking about 1 gigabit, so the actual amount of different videos you can push on a single gigabit is going to be minimal regardless.
True, false and false. so you need experience.
And spindles HGST Ultrastar can actually do it - we're talking about mostly sequential reads and reading very large chunks of data (because videos tend to be more than a megabyte).
Even if you do HLS which is purely chunks of 0.5 to 2-3 megabyte, spinning disks would be perfectly fine.>
You are basically outdated. and do not even know what size of regular video files are these days.
do you know what 480p 720p 1080p 4k is?
And I can't even imagine the number of concurrent connection when you serving few megabytes videos on 18gbps box that has HGST 'spindle' hard disk in it?
In case it's not possible to do it - then just paste 30-40 minutes of "iostat -x -m 1" on LET so we can see that your iops and sequential reads exceeds what a HGST Ultrastar or two can pull off.
>
yeah it might pull 4000-6000 iops and the server hang happen, even in raid 0.
so please go ahead and experience it.
It's video streaming mate :-) Not rocket science.
nothing is science but you just need the real-world experience and realize, to prove your outdated science.
You mean experience as in, working on a CDN focusing on video streaming and doing 600Gbps+ traffic?
Your biggest workload is 1 gigabit, my biggest is 600Gbps+ :-) So I must have 600x more experience than you or?>
From above statement, I don't believe you are doing video streaming right.
and that was just an example from one of my box with 1gbps line when I was into streaming.
And only 600Gbps? that is it?? for CDN streaming?
now I'm serving 3000 IPs to customers and your traffic can't be compared to my business. and from now, I do not have time to spend any more time on you because i'm a business man.
Remeber, all the hardware specifications, data sheet you learned from Google, those should be taken as references, in real-world, your math calculation won't be correct. that's why people still write a "review" after the product specification.
My data I'm sharing with everybody here is all from experiences in fact.
You just didn't realize SSD can go over 10k iops until you experienced it today.
Google isn't god. Experience is.
I can guarantee you that LeaseWeb can't be compared to your small business they're in Ivy league compared to you.
Who has LeaseWeb? and I don't do server hosting business. I do networking.
I think one thing many tests also fail to do, is separation between hot and cold data. Because tiered storage systems are complex things to test. It's totally useless to make quick test where you write and read something. Of course that's fast, it could be served directly from RAM for reads. And writes can be also acknowledged into write log.
I've seen test after test fail with this. And most of tests people recommend to run, absolutely and totally fail measuring the cold data read performance. Which is usually the problem. At least the NVMe / Ceph (what this thread is about) systems usually serve the hot reads fast. But guess what happens when you do cold random read test.
We do use dedicated database servers and those go easily over 200k IOPS on read and even 10k+ IOPS with single thread. And that's production use, not just benchmarking.
Doing 4k:4k fio tests doesn't give a real world perspective - it's fine for having a baseline, that you then run 1 month later to see if stuff is still shit or not.
I'm sorry - I forgot that most people do not have 128-256 gigabyte of memory in a box :') my mistake - this is LET.
If you do HLS streaming (which is quite common), you serve small chunks as you probably know - these will get loaded into memory automatically by the file system cache, if files in linux is frequently accessed, they will stay in memory for quite some time - so in case of HLS, the only disk IO you'll mostly do, comes from reading new HLS chunks, you might have a few system calls to the disk anyway from time to time, but it will be minimal.
If you're serving larger files (480, 720, 1080, 1440 or 4k), same thing will happen - you'll be able to have less files in memory - no doubt, but the OS is smart enough to put "hot" content in places to decrease your disk IO - you can also optimize this even further using sysctl settings, to store even more in memory when possible.
You can verify that every (decent) OS has a file system cache:
sync; echo 3 > /proc/sys/vm/drop_caches; time tar cf blat.tar directory; rm blat.tar; tar cf blat.tar directory
The second tar will be a lot faster due to the fact the file system cache kicked in during the first tar and put as much of the data into memory as possible.
Same goes with video streaming.
But.. You said earlier you do video streaming at 1Gbps :-D
I do have real-world experience - but anyway.
What is false about my statement? You're talking about non outdated documents (so 720p, 1080p, 4k is), we both know that those 3 formats have significant higher bitrate than 240p.
If you do a bit of calculations and take decent quality 720p or 1080p videos the bitrate will be something like 3-5 megabit/s for 720p, 6-8 megabit/s for 1080 and 4k will be 30 megabit/s+
Now let's do some numbers, you do 1 gigabit:
720p: 1000 / 4 = 250 viewers
1080p: 1000 / 7 / 142 viewers
4k: 1000 / 30 = 33 viewers
So on a single gigabit with a decent bitrate you can do a maximum of 250 concurrent users when doing 720p streaming (thus we assume that clients will stream at the exact bitrate, and not do any buffering thus causing a slight congestion on the link).
Ideally you wanna balance your traffic based on URI (video file, chunks etc), to maximize the optimal throughput - it makes no sense to serve let's say 1000 megabit/s of a video over 10 servers if you can put it on one, it's simply stupid to do so. In case a box dies, the ring calculations will change and the streaming will get allocated to another box.
This way you get the most of of your hardware, and actually improves the overall performance of your streams (because you start utilizing buffers and system cache).
It's quite common to have your chunks about 3-5 seconds, so for a 1080p chunk with decent quality would be 3-3.5 megabyte
There's not many people doing 4k HLS streaming these days - but anyway.
Point is, my numbers actually makes sense :-D
If 1080p HLS chunks, 18 gigabit (18000 megabit) is 2571 concurrent connections, pretty easy math, every 4 seconds we'd have to load a new chunk, so we read 3.5 megabyte every 4 seconds for a single 1080p video.
A single HGST Ultrastar will do roughly 180 meg/s sequential reads with that size of files - so you can also figure out pretty easy how many videos you can serve off a single disk - put a few disks (with no raid) in a box - and you can easily hit quite a decent amount of IO - but since we're serving the files primarily from memory anyway, it won't be bad.
Now if you were to stream huge MP4 files of an array, if using the mp4 module in nginx for example it will seek perfectly fine within the file by reading the moov atom - this also means IO actually won't be bad - io will be worse due to the fact the system will try to throw the file into the file system cache - and if you have a large amount of videos, then you start hitting the limits of your memory.
But for doing a single gigabit - sure, a single HGST disk will be enough - especially since we're talking 720p+ content - actually it makes you even more stupid since the stream is at a higher bitrate.
If you do 4-6k iops at 1 gigabit/s 720p+ streams, then you're clearly doing something wrong, even if you serve 100's of 720p videos over a single gigabit connection.
If you use a decent webserver such as nginx, it will as much as it can to use sendfile will happily read at 1.6 megabyte, as you can see here, so you can read 1.6 megabyte of data in pretty much no io.
sendfile(13, 161, [4905514128] => [4907077968], 2147482480) = 1563840 sendfile(13, 161, [4907077968], 2145918640) = -1 EAGAIN (Resource temporarily unavailable) epoll_wait(55, [{EPOLLOUT, {u32=52941328, u64=52941328}}], 512, 2696) = 1 sendfile(13, 161, [4907077968] => [4908706968], 2147483312) = 1629000 sendfile(13, 161, [4908706968], 2145854312) = -1 EAGAIN (Resource temporarily unavailable) epoll_wait(55, [{EPOLLOUT, {u32=52941328, u64=52941328}}], 512, 2657) = 1 sendfile(13, 161, [4908706968] => [4910401128], 2147480424) = 1694160 sendfile(13, 161, [4910401128] => [4910466288], 2145786264) = 65160 sendfile(13, 161, [4910466288], 2145721104) = -1 EAGAIN (Resource temporarily unavailable) epoll_wait(55, [{EPOLLOUT, {u32=52941328, u64=52941328}}], 512, 2616) = 1 sendfile(13, 161, [4910466288] => [4912095288], 2147482384) = 1629000 sendfile(13, 161, [4912095288], 2145853384) = -1 EAGAIN (Resource temporarily unavailable) epoll_wait(55, [{EPOLLOUT, {u32=52941328, u64=52941328}}], 512, 2577) = 1 sendfile(13, 161, [4912095288] => [4913724288], 2147483592) = 1629000 sendfile(13, 161, [4913724288], 2145854592) = -1 EAGAIN (Resource temporarily unavailable) epoll_wait(55, [{EPOLLOUT, {u32=52941328, u64=52941328}}], 512, 2536) = 1 sendfile(13, 161, [4913724288] => [4915288128], 2147480704) = 1563840 sendfile(13, 161, [4915288128], 2145916864) = -1 EAGAIN (Resource temporarily unavailable) epoll_wait(55, [{EPOLLOUT, {u32=52941328, u64=52941328}}], 512, 2494) = 1 sendfile(13, 161, [4915288128] => [4917047448], 2147481536) = 1759320Hæ - no.
Maybe it's the other way around?
Yes - I also said that - maybe read.
So, let's see - 1 gigabit is 1000000 kilobit, you - you stream to 3000 IPs, that means 333.33 kilobit per IP - so we're effectively on low quality 240p bitrate - so why all the talk about 480, 720, 1080 and 4k videos when you use "outdated documents" as you put it.
Why not? Because you do 600Gbps+?
And hi business man ;-) nice to meet you.
Exactly - do not look at the data sheets, and do not perform (fio) tests that are similar to what has been done on the data sheets.
I advise you against sharing experience you don't have - it makes everything so silly.
SSD can go over 10k iops, I never said otherwise - I said it can also be below 10k iops - depending on your workload :-) Again.. read.
Lol, that comment proves the stupidity.
Sure - but depending on your workload, you can also experience less than 10k iops, even on prem SSDs - it always depends on what kind of workloads you throw at a disk or a set of disks.
Guys,
You‘re freaking me out.
Can someone please tell me now which is faster and by what order of magnitude:
Commodore datasette
Sorry, no. Commodore datasette is too fast. It overwhelms the hayes 9.6 k modem!