Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


cheapest way/servers for streaming service
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

cheapest way/servers for streaming service

afnafn Member

Hi There,

We are hosting a private streaming service for a specific archive (data does not grow/increase much). Our storage space is around 1.5 TB and we have a lot of visitors, and the numbers keeps increasing like crazy.
At first, we were using two small dedis from online.net with round-robin DNS and mirroring the content on both servers, but the approach was not so easy to use and we gave up on it when cloudflare kicked us out (their CDN can’t be used for video). Now we are using 1 single server (i7-2600, 6TB HDD) but the server and the drives can’t keep up with the demand.

Also, some of the uploads are by users and some of them of them may violate DMCA, we do not refuse to comply in theory, but we would like to avoid the trouble so we must have our streaming service protected with proxy. Currently we have the server behind a couple of VPSes to hide the real IP and we are using round robin DNS so each user gets directed to a VPS proxy randomly but all VPSes redirect to the same server. The server always has at least 100 people connected to stream, and doing 200+ TB of traffic monthly.

What are the cheapest solutions/improvements we can do to improve the watching experience a bit while remaining on budget (currently we pay around 40€ for the dedi+proxies)? replace by multiple small servers? 1 single better server? other tweaks?

Thanks in advance

Thanked by 1devp
«1

Comments

  • I think you will interested @yoursunny 's NDN architecture

    Thanked by 2yoursunny Durs
  • 40 euro and 200 TB traffic + 1,5 TB storage good luck finding a cheaper option.

    Thanked by 1Durs
  • Nothing

  • deankdeank Member, Troll

    Nada

  • Cheap, reliable bandwidth for streaming, Ignore DCMA?
    Take a look at some illegal movies /jav video sites. What hosting and technology they used, seem fit for your requirements.

  • 250Mbps is not enough for such usage with 100+ streams at once. They need to be buffered ahead of time (If bitrate is 2Mbps then server needs to send at least 3Mbps)

  • AXYZEAXYZE Member
    edited October 2021

    @afn said:
    Hi There,

    We are hosting a private streaming service for a specific archive (data does not grow/increase much). Our storage space is around 1.5 TB and we have a lot of visitors, and the numbers keeps increasing like crazy.
    At first, we were using two small dedis from online.net with round-robin DNS and mirroring the content on both servers, but the approach was not so easy to use and we gave up on it when cloudflare kicked us out (their CDN can’t be used for video). Now we are using 1 single server (i7-2600, 6TB HDD) but the server and the drives can’t keep up with the demand.

    Also, some of the uploads are by users and some of them of them may violate DMCA, we do not refuse to comply in theory, but we would like to avoid the trouble so we must have our streaming service protected with proxy. Currently we have the server behind a couple of VPSes to hide the real IP and we are using round robin DNS so each user gets directed to a VPS proxy randomly but all VPSes redirect to the same server. The server always has at least 100 people connected to stream, and doing 200+ TB of traffic monthly.

    What are the cheapest solutions/improvements we can do to improve the watching experience a bit while remaining on budget (currently we pay around 40€ for the dedi+proxies)? replace by multiple small servers? 1 single better server? other tweaks?

    Thanks in advance

    Hetzner server auction with 32/64GB of ram (kernel will page cache frequently used files, HDD can be enough), but you need to proxy it by something else ex BuyVM, because Hetz is not happy about such stuff. They are unmetered

    Netcup root server has 120TB bandwidth limit, after that it slows down to 200Mbps.

    OVH VPS with their block storage or some other block storage csn be good option too. Use boot volume NVMe as cache.

    Check scaleway dedis too.

  • afnafn Member

    @eris said:
    40 euro and 200 TB traffic + 1,5 TB storage good luck finding a cheaper option.

    Never said I am looking for cheaper! could you please quote where I said I am not happy paying 40€? I clearly said "looking for cheapest" improvement we can do in other terms, improving the situation without having to add 400€/monthly to the bill.

    @hanoi said: Take a look at some illegal movies /jav video sites. What hosting and technology they used, seem fit for your requirements.

    Most of them do not self host their videos, they have one (or a couple of) personal server (s) for their use + uploads and some batch scripts to keep reuploading videos to 3rd party sites via APIs when links are DMCed and replaces links in the streaming frontend.

  • Depending on how the load is distributed over current video collection. If it is random "request" it might be harder to get it working.

    But in the past I had an 2 "Master" that had direct access with the "storage" + Few "larger" nodes who would pick up the the load bypassed an certain treshold for after a video hit an certain number of fews in the last x min. the larger "nodes" had a local nvme cache of a few 100 gb and would handle most traffic. The slower "storage" servers only took the traffic for the majority random requests.

    It worked great for a long time after the sales declined but was more expensive then 400 euro / month..

    The colocation would already be 900 euros for + 20TB of video's

  • afnafn Member
    edited October 2021

    @AXYZE said: Hetzner server auction with 32/64GB of ram (kernel will page cache frequently used files, HDD can be enough), but you need to proxy it by something else ex BuyVM, because Hetz is not happy about such stuff. They are unmetered

    Literally the current solution solution we are using, i7-2600 with HDD. we have several proxies because VPS is shared and to avoid being noisy neighbors we have multiple ones with different providers, some of them are with buyVM.

    I hate OVH. Scaleway is not the best value for money ratio, hetzner and leaseweb via resellers like walkerservers are so far our go-to for. Thanks for suggesting netcup, They seems to be good enough for our use case.

    I am hesitating a bit if should upgrade to one much better server or mirror the content to another cheap server and have 2 mediocre servers serving files to users with some round-robin DNS.

    Thanked by 1nyamenk
  • @afn said: netcup

    Germany is one of those countries you'd never want to go for if you are expecting DMCA issues

  • vovlervovler Member
    edited October 2021

    1) Get BuyVM LU for 3.5$/m + 2TB Slab for $10/m

    2) Get 10x Stardust Scaleway instances for about $2/m each -> $20/m, install debian + nginx on all of them (you only get 10GB of storage, you cant just install centos with cpanel). Enable reverse proxy cache and reverse proxy only your video files.

    3) Have a separate subdomain for the media files, something like video.domain.com

    The (1) would be your main server, store your files there and thats where your users connect to. The (3) subdomain would round robin the 10x IPs from (2). You can easily add more (2) instances as you require more edge bandwidth. If (1) is using too much bandwidth, increase the disk size of (2) and the reverse proxy cache max size.

    Thanked by 1devp
  • afnafn Member
    edited October 2021

    @vovler thanks a lot for the suggestion! We already have the main domain /frontend for site.com running on a separate vps and the video at videos.domain.com. Currently we use the proxies for videos.domain.com we don't need it for the main website.

    Any reason for picking Stardust Scaleway in particular? I like your idea, but there is one tiny problem (if I understand correctly), the scaleways servers IPs will be exposed, which means DMCA to scaleway, not sure how they handle complaints. We only got 3 during 1 year, but I expect to start getting much more as we get more visitors. I am just thinking of doing your idea with a mix of 2$/3.5$ buyVM instances instead and enable cache on them.

    Also, I know netcup (germany) is not ideal for dmca, but It will be behind a proxy if I use their services, so it should be fine

  • AXYZEAXYZE Member
    edited October 2021

    @vovler said:

    @afn said: netcup

    Germany is one of those countries you'd never want to go for if you are expecting DMCA issues

    That's why he should proxy data so if anything happens only proxy without files will get terminated. It doesn't matter where he hosts, he should use proxy for this kind of stuff always.

    Get BuyVM LU for 3.5$/m + 2TB Slab for $10/m

    You know that BuyVM traffic is fair share right? It's 100Mbps for 4GB and you're recommending 1GB variant (so 25Mbps for that one) for someone who needs 400Mbps+ (and that's VERY conservative calculation, if he is sending HD videos at around 3Mbps bitrate). If nothing changed from 2020 then he will get throttled really hard for it. I search online and I dont see any mention from Francisco that he removed this limit, so probably you're not aware that it exists. His usage is not burst, videos are long and needs to buffer all the time.

    2) Get 10x Stardust Scaleway instances for about $2/m each

    If system takes 4GB of space then you can use 6GB of remaining space. Even if these movies are low-bitrate rips 1GB each then you can place maximum of 6 on single server. So maximum 60 videos. But he already has 100+ concurrent users... ? Even if this video is cached then there's 10% chance that you will connect to this instance, because you want him to buy 10 instances. In what situation it is supposed to help?

    If people are watching same movie then it is cached on RAM, linux page caches it by default. These 10x instances wont help with anything there, they could only help if someone would want to access video that is not cached on main machine, but they wont help there because they have just 6GB of useful space.

    On top of that, you can get up to 2 Stardust instances per your details.

    Enable reverse proxy cache and reverse proxy only your video files.

    I won't even get started on it. Have you once did this similar like this and you know how nginx will work with caching 1GB objects? Especially when 10 nginx instances want to do it from 1 server with HDD?

    Thanked by 1devp
  • AXYZEAXYZE Member
    edited October 2021

    @afn said:
    @vovler thanks a lot for the suggestion! We already have the main domain /frontend for site.com running on a separate vps and the video at videos.domain.com. Currently we use the proxies for videos.domain.com we don't need it for the main website.

    Any reason for picking Stardust Scaleway in particular? I like your idea, but there is one tiny problem (if I understand correctly), the scaleways servers IPs will be exposed, which means DMCA to scaleway, not sure how they handle complaints. We only got 3 during 1 year, but I expect to start getting much more as we get more visitors. I am just thinking of doing your idea with a mix of 2$/3.5$ buyVM instances instead and enable cache on them.

    Also, I know netcup (germany) is not ideal for dmca, but It will be behind a proxy if I use their services, so it should be fine

    Just use netcup as backend. They are solid, they will give you 120TB traffic and after that they throttle to 200Mbps. Backend server is a lot more important, as all your files will be there.
    If storage space isnt enough then place less frequently watches movies on some other service, other server like this one https://contabo.com/en/storage-vps/ or use this https://www.netcup.eu/vserver/storagespace.php
    Plenty of options.

    Proxy server can be replaced in like 10 minutes, you don't need to care that much about it as long as they wont forward info about you because of DMCA. If they will terminate you then its not big deal, get another one. If backend serer will be terminated then its huge problem for you.

  • defaultdefault Veteran
    edited October 2021

    Thanked by 1dahartigan
  • afnafn Member

    @AXYZE THANK YOU!

    @AXYZE said: You know that BuyVM traffic is fair share right? It's 100Mbps for 4GB and you're recommending 1GB variant .... If nothing changed from 2020 then he will get throttled really hard for it.

    It's already the case, our instances with buyVM already get throttled frequently, we have multiple proxies so it helps a bit reducing load on a single one, but still.

    I am thinking :

    • keep frequently accessed files on a good netcup root server (since it's fast but has small storage) [or should I consider their storage servers with SAS drives?]
    • less accessed files on the hetzner slow dedi we already have
    • Set up our proxies on several small VPSes (user gets redirected to one of the proxies randomly with round robin DNS) and each proxy uses nginx to get the file from the main server (I will keep both servers ofc behind proxies.)

    I know how to make my proxies request the file from the correct server (netcup/hetzner) based on url. May I ask what VPS/cloud instances do you suggest for proxy? and how many should I run?

    Thanks

    Thanked by 1devp
  • @afn said:
    @AXYZE THANK YOU!

    @AXYZE said: You know that BuyVM traffic is fair share right? It's 100Mbps for 4GB and you're recommending 1GB variant .... If nothing changed from 2020 then he will get throttled really hard for it.

    It's already the case, our instances with buyVM already get throttled frequently, we have multiple proxies so it helps a bit reducing load on a single one, but still.

    I am thinking :

    • keep frequently accessed files on a good netcup root server (since it's fast but has small storage) [or should I consider their storage servers with SAS drives?]
    • less accessed files on the hetzner slow dedi we already have
    • Set up our proxies on several small VPSes (user gets redirected to one of the proxies randomly with round robin DNS) and each proxy uses nginx to get the file from the main server (I will keep both servers ofc behind proxies.)

    I know how to make my proxies request the file from the correct server (netcup/hetzner) based on url. May I ask what VPS/cloud instances do you suggest for proxy? and how many should I run?

    Thanks

    I have no idea which DMCA-ignored VPS will be the best for proxy. Try to make separate post about it, maybe you'll get good offer from some provider.

  • afnafn Member
    edited October 2021

    okay, thanks :smiley:

    Appreciate your, and everyone else's tips

  • @hanoi said:
    Cheap, reliable bandwidth for streaming, Ignore DCMA?
    Take a look at some illegal movies /jav video sites. What hosting and technology they used, seem fit for your requirements.

    not really because look at gogoanime they were literally using google cloud a moment ago xD

  • lifehost360_comlifehost360_com Member, Host Rep

    cheaper than 40€ / Month?? seriously??

    we´re hosting a VoD-portal for governmental critics news at our Helsinki DC... The customer does approx. 200TB of traffic a month and has to pay 1400€ monthly just for the streaming- and apiservers and to get access to 10G Uplink. (it´s a bit more necessary to serve their up to 30k users...)

    good luck finding any competitor.

  • lifehost360_comlifehost360_com Member, Host Rep
    edited October 2021

    @afn said:

    I am thinking :

    • keep frequently accessed files on a good netcup root server (since it's fast but has small storage) [or should I consider their storage servers with SAS drives?]

    SAS as well as any HDD is too slow for your needs. You´ll need to have at very least SATA SSD. even when SAS is supported with 12G - the i/o kills you. A HDD only has up to 80i/o per seconds, normally usable are 50-60IOPS per disc.

    I know how to make my proxies request the file from the correct server (netcup/hetzner) based on url. May I ask what VPS/cloud instances do you suggest for proxy? and how many should I run?

    you might think about a varnish cash, hosted on small VPS with SSD

  • sandozsandoz Veteran
    edited October 2021

    Here you have..
    https://xor.sc (not good for people with Panic)

  • You: We do 200+TB/Month, we want more, more and more!

    Hetzner: Bruh, you are only paying 25 euros/month.

    Me: Recommend @Clouvider

  • yoursunnyyoursunny Member, IPv6 Advocate

    @chocolateshirt said:
    I think you will interested @yoursunny 's NDN architecture

    Global NDN testbed has over 30 nodes around the world, each with 1Gbps uplink.
    Software is here: https://github.com/yoursunny/NDNts-video-server
    You'll have to reencode all the videos into DASH format though.

  • @afn said: improvements we can do to improve the watching experience a bit while remaining on budget

    This is hard thing to do. Really complex. I think there are a lot of experts on LET, who will offer much better solution, but if I'm forced to do something with streams, my algorithm will be next:

    1. reduce quality of video content by using modern codecs and reducing bitrate.
    2. in video player that play video, add default option to play 360p, with switcher to select better video quality on request. This is must have feature.
    3. Throttle bw connection per IP to max video stream bitrate. Because some people will open 2-3-5-10 tabs or download manager, and will consume all bw, while others will suffer a lot because of no free channel.
    4. Docker is must-have. Alternatively need to take a look on kubernets and dynamic horizontal vps instances deployment on the fly when loading increased. The main idea: keep as less as you can computing power, and increase horizontally your instances when load increased. There are a lot of providers with API for doing that: DigitalOcean, Vultr, OVH, Linode, etc. But there must be tons of scripts and a lot of coding, and configs depends on the providers API answers...
  • afnafn Member
    edited October 2021

    @lifehost360_com said:
    cheaper than 40€ / Month?? seriously??

    we´re hosting a VoD-portal for governmental critics news at our Helsinki DC... The customer does approx. 200TB of traffic a month and has to pay 1400€ monthly just for the streaming- and apiservers and to get access to 10G Uplink. (it´s a bit more necessary to serve their up to 30k users...)

    good luck finding any competitor.

    @lifehost360_com When the hell did I say I want cheaper than 40€? someone else made the same comment as you. I also replied to them in:
    https://www.lowendtalk.com/discussion/comment/3291008/#Comment_3286458

    I said what we are paying but never said we are not ready to pay more! The whole point is: I am saying I have a cheap solution that doesn't work well so I am looking for a not so costly upgrade (500€+ !) that works better

    @Hayashima

    Same thing, read my reply to lifehost. I am literally saying a 25€ hetzner server does not do the job, so what is better

  • afnafn Member

    @lifehost360_com said:

    you might think about a varnish cash, hosted on small VPS with SSD

    thanks for the tip. how is that different from activating the nginx cache on our reverse proxy?

    @yoursunny said: Global NDN testbed has over 30 nodes around the world, each with 1Gbps uplink.

    Software is here: https://github.com/yoursunny/NDNts-video-server
    You'll have to reencode all the videos into DASH format though.
    reencode of all videos to DASH might be a very time consuming task for existing videos. But still worth considering, I will look more into it, Thanks!

    @sandoz said:
    Here you have..
    https://xor.sc (not good for people with Panic)

    That is exactly what I was looking for, fits perfectly for my budget their price for 100TB is very affordable, it is a good upgrade without breaking the bank. (srsly, do they even have any clients!?)

  • afnafn Member
    1. reduce quality of video content by using modern codecs and reducing bitrate.

    Already doing it.

    1. in video player that play video, add default option to play 360p, with switcher to select better video quality on request. This is must have feature.

    Nice trick some people might be lazy to switch, also it is worth noting that most videos are 480p already (150-200MB).

    1. Throttle bw connection per IP to max video stream bitrate. Because some people will open 2-3-5-10 tabs or download manager, and will consume all bw, while others will suffer a lot because of no free channel.

    Already doing it.

    1. Docker is must-have. Alternatively need to take a look on kubernets and dynamic horizontal vps instances deployment on the fly when loading increased. The main idea: keep as less as you can computing power, and increase horizontally your instances when load increased. There are a lot of providers with API for doing that: DigitalOcean, Vultr, OVH, Linode, etc. But there must be tons of scripts and a lot of coding, and configs depends on the providers API answers...

    Aside from the difficulty of writing a good working script that does the job. I need to study more our load pattern, if most traffic happens during weekend only for example, this might be a good solution, but the risk of increasing my instances on the fly is that if it is not hard-limited in the script, you may end up paying 1000€ bill at the end of the month.

    To be realistic, I think we are ready to double what we're paying (so around ~80€ is our current budget) and I don't think the idea of increasing instances on demand fits that budget...

Sign In or Register to comment.