I wrote a shell script that turns spare VPS into a CDN node in one click

Lutung

Please specify your private key file location (e.g., /etc/certs/qing.su.key):
/etc/certs/abcd.com.key
Please specify your certificate file location (e.g., /etc/certs/qing.su.crt):
/etc/certs/abcd.com.crt
Is your certificate chained? (i.e., are CA-certficates already included in your certificate file?) [Y/N]:
N
Please specify your CA-certificates file location (e.g., /etc/certs/qing.su.ca-bundle):

Configuring SSL certificates for xxx.xxx.xxx.xxx...
cp: missing destination file operand after '/etc/trafficserver/ssl/abcd.com.ca.crt'
Try 'cp --help' for more information.

I got this problem too.

WebProject

@nocloud said:

@WebProject said:

@umi said:
Huge script from China that needs sudo privilegies. I'll pass

Very small script and open source without any odd code, see yourself:
https://github.com/Har-Kuun/OneClickCDN/blob/master/OneClickCDN.sh

** Based on your comments you can work for BBC TV.**

what do you mean by this statement exactly?

its open source code so you be able to check the code for any backdoors. Personally will not trust any news on BBC TV as they are full of lies!

Dazzle

Man you're awesome. Testing it and work perfectly.

JasonM

@dedipromo said: Ubuntu 20.04 LTS 64 bit, or Debian 10 (experimental)

do the entire fleet of vps I have require same Ubuntu 20.4 64-bit? Or can work with different versions?

Boogeyman

@JasonM said: do the entire fleet of vps I have require same Ubuntu 20.4 64-bit? Or can work with different versions?

RHEL based will work too. I have checked the code.

Void

Interesting. Will it work for NAT VPS?

kkrajk

@WebProject said: BBC TV

BBC ?

trycatchthis

What is the name of the self hosted CDN package you are using?

ralf

Looking at the script, this seems like the relatively simple part of setting up a caching proxy server, for which there are already lots of tutorals online. I'm actually more interested in how you implement the harder things in this set-up, specifically:

If you have multiple machines, any of which could serve your web traffic, how do you handle letsencrypt certificate renewal when you don't know which one will end up receiving the verification check. You could have a reverse proxy on each machine that knows to forward to the machine currently renewing, but how to set that up is the interesting thing, as the default certbot manages all this stuff for you by patching apache/nginx configuration for you, but only works on a single machine. I can see multiple options, e.g. use DNS, have one server create the cert and ssh it around, a complicated proxy forwarding system, but nowhere that actually recommends a best practice.

How to set up GeoDNS to go to the best of your local caches. Ideally, we'd be able to implement this ourselves for free, but searching for GeoDNS only turns up commercial services. In theory, this is hard, as a lot of people will be using Google's DNS, so presumably individual nodes should also be figuring out where customers are coming from and redirecting them to their best server. Or do you just round-robin DNS records and hope for the best?

dedipromo

@JasonM said:

@dedipromo said: Ubuntu 20.04 LTS 64 bit, or Debian 10 (experimental)

do the entire fleet of vps I have require same Ubuntu 20.4 64-bit? Or can work with different versions?

They can be different OS, no need to be the same, as long as they can run this script.

dedipromo

@jmaxwell said:
Interesting. Will it work for NAT VPS?

As long as you have Port 80/443 access then it should work. One thing to take into consideration: NAT VPS usually comes with very throttled CPU, which might not be enough to finish the compilation step.

dedipromo

@trycatchthis said:
What is the name of the self hosted CDN package you are using?

Apache Trafficserver.

dedipromo

@ralf said:
Looking at the script, this seems like the relatively simple part of setting up a caching proxy server, for which there are already lots of tutorals online. I'm actually more interested in how you implement the harder things in this set-up, specifically:

If you have multiple machines, any of which could serve your web traffic, how do you handle letsencrypt certificate renewal when you don't know which one will end up receiving the verification check. You could have a reverse proxy on each machine that knows to forward to the machine currently renewing, but how to set that up is the interesting thing, as the default certbot manages all this stuff for you by patching apache/nginx configuration for you, but only works on a single machine. I can see multiple options, e.g. use DNS, have one server create the cert and ssh it around, a complicated proxy forwarding system, but nowhere that actually recommends a best practice.

How to set up GeoDNS to go to the best of your local caches. Ideally, we'd be able to implement this ourselves for free, but searching for GeoDNS only turns up commercial services. In theory, this is hard, as a lot of people will be using Google's DNS, so presumably individual nodes should also be figuring out where customers are coming from and redirecting them to their best server. Or do you just round-robin DNS records and hope for the best?

Thanks for the suggestions. For the first part, the script has an option to backup all configuration on one node and restore it on another node, so you can set up Let's Encrypt on one node, backup everything and restore on other nodes, and the certificates will now work on all nodes.

For the second part about the GeoDNS, I'll have to think about how to implement this.

dedipromo

@Lutung said:
Please specify your private key file location (e.g., /etc/certs/qing.su.key):
/etc/certs/abcd.com.key
Please specify your certificate file location (e.g., /etc/certs/qing.su.crt):
/etc/certs/abcd.com.crt
Is your certificate chained? (i.e., are CA-certficates already included in your certificate file?) [Y/N]:
N
Please specify your CA-certificates file location (e.g., /etc/certs/qing.su.ca-bundle):

Configuring SSL certificates for xxx.xxx.xxx.xxx...
cp: missing destination file operand after '/etc/trafficserver/ssl/abcd.com.ca.crt'
Try 'cp --help' for more information.

I got this problem too.

Looks like you did not type in the CA chain certs file. If you don't have the CA chain, then the chain should already be included in the certificate file, the you'll need to type "Y" instead of "N" for the third question.

trycatchthis

@dedipromo said:

@trycatchthis said:
What is the name of the self hosted CDN package you are using?

Apache Trafficserver.

Do you have docker support in mind?

Daniel15

@ralf said: , but searching for GeoDNS only turns up commercial services.

For self-hosted, PowerDNS can do GeoIP via Lua records (don't use the legacy YAML syntax). It can pick the one closest to the user that's up (so if a server is down, it won't return that server's IP).

Round Robin is not ideal for CDN nodes as users may get a node that's very far from them.

An alternative that doesn't need any fancy DNS configuration is to use separate subdomains (eg. us.example.com, uk.example.com) and update your app to render all static file URLs with the right subdomain. This means you can't use the CDN for the initial load though, but this is the more common use case for CDNs anyways (as the home page is often dynamic).

@ralf said: how do you handle letsencrypt certificate renewal when you don't know which one will end up receiving the verification check.

Two options I can think of:

1. Renew certs once and then distribute the same cert to each server. Distribute them using whatever method you use to distribute other config files (I use Ansible for this).
2. Use DNS instead. Just be sure to configure each node to renew at different times so they're not all updating the DNS record at the same time. I use acme-dns for hosting the Let's Encrypt DNS records, so that my servers don't need an API key with full access to the regular DNS server.

Lutung

@dedipromo said:

@Lutung said:
Please specify your private key file location (e.g., /etc/certs/qing.su.key):
/etc/certs/abcd.com.key
Please specify your certificate file location (e.g., /etc/certs/qing.su.crt):
/etc/certs/abcd.com.crt
Is your certificate chained? (i.e., are CA-certficates already included in your certificate file?) [Y/N]:
N
Please specify your CA-certificates file location (e.g., /etc/certs/qing.su.ca-bundle):

Configuring SSL certificates for xxx.xxx.xxx.xxx...
cp: missing destination file operand after '/etc/trafficserver/ssl/abcd.com.ca.crt'
Try 'cp --help' for more information.

I got this problem too.

Looks like you did not type in the CA chain certs file. If you don't have the CA chain, then the chain should already be included in the certificate file, the you'll need to type "Y" instead of "N" for the third question.

Yeah that I have made a mistake too.

painfreepc

sorry i am a little green,

my cdn cache server has 150gb storage, 130gb is used for cache, origin server is using 700GB (soon to be using 1.5 TB) of files,
so how is cdn handing that, is old request cache dropped as new request come in?

Neoon

@painfreepc said:
sorry i am a little green,

my cdn cache server has 150gb storage, 130gb is used for cache, origin server is using 700GB (soon to be using 1.5 TB) of files,
so how is cdn handing that, is old request cache dropped as new request come in?

It likely will kill existing cached files to make space for the new requested ones.
Which will likely drag the performance down, due to low local disk space.

I would suggest at least half of that on the CDN Server.