All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
cheapest way/servers for streaming service
Hi There,
We are hosting a private streaming service for a specific archive (data does not grow/increase much). Our storage space is around 1.5 TB and we have a lot of visitors, and the numbers keeps increasing like crazy.
At first, we were using two small dedis from online.net with round-robin DNS and mirroring the content on both servers, but the approach was not so easy to use and we gave up on it when cloudflare kicked us out (their CDN can’t be used for video). Now we are using 1 single server (i7-2600, 6TB HDD) but the server and the drives can’t keep up with the demand.
Also, some of the uploads are by users and some of them of them may violate DMCA, we do not refuse to comply in theory, but we would like to avoid the trouble so we must have our streaming service protected with proxy. Currently we have the server behind a couple of VPSes to hide the real IP and we are using round robin DNS so each user gets directed to a VPS proxy randomly but all VPSes redirect to the same server. The server always has at least 100 people connected to stream, and doing 200+ TB of traffic monthly.
What are the cheapest solutions/improvements we can do to improve the watching experience a bit while remaining on budget (currently we pay around 40€ for the dedi+proxies)? replace by multiple small servers? 1 single better server? other tweaks?
Thanks in advance
Comments
I think you will interested @yoursunny 's NDN architecture
40 euro and 200 TB traffic + 1,5 TB storage good luck finding a cheaper option.
Nothing
Nada
Cheap, reliable bandwidth for streaming, Ignore DCMA?
Take a look at some illegal movies /jav video sites. What hosting and technology they used, seem fit for your requirements.
try sys
https://www.lowendtalk.com/discussion/174282/so-you-start-flash-sale#latest
250Mbps is not enough for such usage with 100+ streams at once. They need to be buffered ahead of time (If bitrate is 2Mbps then server needs to send at least 3Mbps)
Hetzner server auction with 32/64GB of ram (kernel will page cache frequently used files, HDD can be enough), but you need to proxy it by something else ex BuyVM, because Hetz is not happy about such stuff. They are unmetered
Netcup root server has 120TB bandwidth limit, after that it slows down to 200Mbps.
OVH VPS with their block storage or some other block storage csn be good option too. Use boot volume NVMe as cache.
Check scaleway dedis too.
Never said I am looking for cheaper! could you please quote where I said I am not happy paying 40€? I clearly said "looking for cheapest" improvement we can do in other terms, improving the situation without having to add 400€/monthly to the bill.
Most of them do not self host their videos, they have one (or a couple of) personal server (s) for their use + uploads and some batch scripts to keep reuploading videos to 3rd party sites via APIs when links are DMCed and replaces links in the streaming frontend.
Depending on how the load is distributed over current video collection. If it is random "request" it might be harder to get it working.
But in the past I had an 2 "Master" that had direct access with the "storage" + Few "larger" nodes who would pick up the the load bypassed an certain treshold for after a video hit an certain number of fews in the last x min. the larger "nodes" had a local nvme cache of a few 100 gb and would handle most traffic. The slower "storage" servers only took the traffic for the majority random requests.
It worked great for a long time after the sales declined but was more expensive then 400 euro / month..
The colocation would already be 900 euros for + 20TB of video's
Literally the current solution solution we are using, i7-2600 with HDD. we have several proxies because VPS is shared and to avoid being noisy neighbors we have multiple ones with different providers, some of them are with buyVM.
I hate OVH. Scaleway is not the best value for money ratio, hetzner and leaseweb via resellers like walkerservers are so far our go-to for. Thanks for suggesting netcup, They seems to be good enough for our use case.
I am hesitating a bit if should upgrade to one much better server or mirror the content to another cheap server and have 2 mediocre servers serving files to users with some round-robin DNS.
Germany is one of those countries you'd never want to go for if you are expecting DMCA issues
1) Get BuyVM LU for 3.5$/m + 2TB Slab for $10/m
2) Get 10x Stardust Scaleway instances for about $2/m each -> $20/m, install debian + nginx on all of them (you only get 10GB of storage, you cant just install centos with cpanel). Enable reverse proxy cache and reverse proxy only your video files.
3) Have a separate subdomain for the media files, something like video.domain.com
The (1) would be your main server, store your files there and thats where your users connect to. The (3) subdomain would round robin the 10x IPs from (2). You can easily add more (2) instances as you require more edge bandwidth. If (1) is using too much bandwidth, increase the disk size of (2) and the reverse proxy cache max size.
@vovler thanks a lot for the suggestion! We already have the main domain /frontend for site.com running on a separate vps and the video at videos.domain.com. Currently we use the proxies for videos.domain.com we don't need it for the main website.
Any reason for picking Stardust Scaleway in particular? I like your idea, but there is one tiny problem (if I understand correctly), the scaleways servers IPs will be exposed, which means DMCA to scaleway, not sure how they handle complaints. We only got 3 during 1 year, but I expect to start getting much more as we get more visitors. I am just thinking of doing your idea with a mix of 2$/3.5$ buyVM instances instead and enable cache on them.
Also, I know netcup (germany) is not ideal for dmca, but It will be behind a proxy if I use their services, so it should be fine
That's why he should proxy data so if anything happens only proxy without files will get terminated. It doesn't matter where he hosts, he should use proxy for this kind of stuff always.
You know that BuyVM traffic is fair share right? It's 100Mbps for 4GB and you're recommending 1GB variant (so 25Mbps for that one) for someone who needs 400Mbps+ (and that's VERY conservative calculation, if he is sending HD videos at around 3Mbps bitrate). If nothing changed from 2020 then he will get throttled really hard for it. I search online and I dont see any mention from Francisco that he removed this limit, so probably you're not aware that it exists. His usage is not burst, videos are long and needs to buffer all the time.
If system takes 4GB of space then you can use 6GB of remaining space. Even if these movies are low-bitrate rips 1GB each then you can place maximum of 6 on single server. So maximum 60 videos. But he already has 100+ concurrent users... ? Even if this video is cached then there's 10% chance that you will connect to this instance, because you want him to buy 10 instances. In what situation it is supposed to help?
If people are watching same movie then it is cached on RAM, linux page caches it by default. These 10x instances wont help with anything there, they could only help if someone would want to access video that is not cached on main machine, but they wont help there because they have just 6GB of useful space.
On top of that, you can get up to 2 Stardust instances per your details.
I won't even get started on it. Have you once did this similar like this and you know how nginx will work with caching 1GB objects? Especially when 10 nginx instances want to do it from 1 server with HDD?
Just use netcup as backend. They are solid, they will give you 120TB traffic and after that they throttle to 200Mbps. Backend server is a lot more important, as all your files will be there.
If storage space isnt enough then place less frequently watches movies on some other service, other server like this one https://contabo.com/en/storage-vps/ or use this https://www.netcup.eu/vserver/storagespace.php
Plenty of options.
Proxy server can be replaced in like 10 minutes, you don't need to care that much about it as long as they wont forward info about you because of DMCA. If they will terminate you then its not big deal, get another one. If backend serer will be terminated then its huge problem for you.
@AXYZE THANK YOU!
It's already the case, our instances with buyVM already get throttled frequently, we have multiple proxies so it helps a bit reducing load on a single one, but still.
I am thinking :
I know how to make my proxies request the file from the correct server (netcup/hetzner) based on url. May I ask what VPS/cloud instances do you suggest for proxy? and how many should I run?
Thanks
I have no idea which DMCA-ignored VPS will be the best for proxy. Try to make separate post about it, maybe you'll get good offer from some provider.
okay, thanks
Appreciate your, and everyone else's tips
not really because look at gogoanime they were literally using google cloud a moment ago xD
cheaper than 40€ / Month?? seriously??
we´re hosting a VoD-portal for governmental critics news at our Helsinki DC... The customer does approx. 200TB of traffic a month and has to pay 1400€ monthly just for the streaming- and apiservers and to get access to 10G Uplink. (it´s a bit more necessary to serve their up to 30k users...)
good luck finding any competitor.
SAS as well as any HDD is too slow for your needs. You´ll need to have at very least SATA SSD. even when SAS is supported with 12G - the i/o kills you. A HDD only has up to 80i/o per seconds, normally usable are 50-60IOPS per disc.
you might think about a varnish cash, hosted on small VPS with SSD
Here you have..
https://xor.sc (not good for people with Panic)
You: We do 200+TB/Month, we want more, more and more!
Hetzner: Bruh, you are only paying 25 euros/month.
Me: Recommend @Clouvider
Global NDN testbed has over 30 nodes around the world, each with 1Gbps uplink.
Software is here: https://github.com/yoursunny/NDNts-video-server
You'll have to reencode all the videos into DASH format though.
This is hard thing to do. Really complex. I think there are a lot of experts on LET, who will offer much better solution, but if I'm forced to do something with streams, my algorithm will be next:
@lifehost360_com When the hell did I say I want cheaper than 40€? someone else made the same comment as you. I also replied to them in:
https://www.lowendtalk.com/discussion/comment/3291008/#Comment_3286458
I said what we are paying but never said we are not ready to pay more! The whole point is: I am saying I have a cheap solution that doesn't work well so I am looking for a not so costly upgrade (500€+ !) that works better
@Hayashima
Same thing, read my reply to lifehost. I am literally saying a 25€ hetzner server does not do the job, so what is better
thanks for the tip. how is that different from activating the nginx cache on our reverse proxy?
Software is here: https://github.com/yoursunny/NDNts-video-server
You'll have to reencode all the videos into DASH format though.
reencode of all videos to DASH might be a very time consuming task for existing videos. But still worth considering, I will look more into it, Thanks!
That is exactly what I was looking for, fits perfectly for my budget their price for 100TB is very affordable, it is a good upgrade without breaking the bank. (srsly, do they even have any clients!?)
Already doing it.
Nice trick some people might be lazy to switch, also it is worth noting that most videos are 480p already (150-200MB).
Already doing it.
Aside from the difficulty of writing a good working script that does the job. I need to study more our load pattern, if most traffic happens during weekend only for example, this might be a good solution, but the risk of increasing my instances on the fly is that if it is not hard-limited in the script, you may end up paying 1000€ bill at the end of the month.
To be realistic, I think we are ready to double what we're paying (so around ~80€ is our current budget) and I don't think the idea of increasing instances on demand fits that budget...