All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
RTMPS w/ Nginx & Stunnel
ChrisCantwell
Member
In order to fight the forces of darkness I have endeavored to make the unambiguously righteous message of my wildly entertaining live stream and podcast heard by as many people as possible, and this meets no shortage of challenges, as you might expect in a world befallen by so much darkness.
Fortunately I appear to be aided by God to some measure in this, and it also helps that I have some technical skills.
God here is obviously faultless, but sometimes my skills do fail me. I was, after all, not in practice for the better part of three years while enjoying the hospitality of my government, and I hope I might seek the assistance of this upstanding community in solving a problem with the Nginx RTMP module, and Stunnel, which is required to facilitate secure connections with that module.
The Nginx RTMP module, some of you may know, does not support RTMPS. This renders impossible using it directly as a relay to services like Kick.com and Telegram and Entropy. One must use a secure tunnel (S-Tunnel = Stunnel) to facilitate such connections.
Rumble, Odysee, Twitch, DLive, VaughnLive, NimoTV, Trovo, and other platforms I stream to do not have this problem, which tells me that the relay itself works fine.
When streaming to this many platforms, it can put some strain on one's upstream bandwidth, hence my use of the relay server. I also use an HLS output to embed video feeds on my own sites.
I have created three files in /etc/stunnel/conf.d to describe the services I am to use, and (after some troubles figuring this out initially) these load fine and stunnel loads.
root@mailstream:~# systemctl status stunnel4
● stunnel4.service - LSB: Start or stop stunnel 4.x (TLS tunnel for network daemons)
Loaded: loaded (/etc/init.d/stunnel4; generated)
Active: active (running) since Wed 2023-10-11 06:44:15 EDT; 3s ago
Docs: man:systemd-sysv-generator(8)
Process: 4031 ExecStart=/etc/init.d/stunnel4 start (code=exited, status=0/SUCCESS)
Tasks: 2 (limit: 5155)
Memory: 2.5M
CPU: 58ms
CGroup: /system.slice/stunnel4.service
└─4047 /usr/bin/stunnel4 /etc/stunnel/stunnel.conf
Oct 11 06:44:15 mailstream.surrealpolitiks.com stunnel[4045]: LOG5[ui]: Reading configuration from file /etc/stunnel/conf.d/kick.conf
Oct 11 06:44:15 mailstream.surrealpolitiks.com stunnel[4045]: LOG5[ui]: UTF-8 byte order mark not detected
Oct 11 06:44:15 mailstream.surrealpolitiks.com stunnel4[4031]: Starting TLS tunnels: /etc/stunnel/stunnel.conf: started
Oct 11 06:44:15 mailstream.surrealpolitiks.com systemd[1]: Started LSB: Start or stop stunnel 4.x (TLS tunnel for network daemons).
Oct 11 06:44:15 mailstream.surrealpolitiks.com stunnel[4045]: LOG4[ui]: Service [Entropy] needs authentication to prevent MITM attacks
Oct 11 06:44:15 mailstream.surrealpolitiks.com stunnel[4045]: LOG5[ui]: Reading configuration from file /etc/stunnel/conf.d/telegram.conf
Oct 11 06:44:15 mailstream.surrealpolitiks.com stunnel[4045]: LOG5[ui]: UTF-8 byte order mark not detected
Oct 11 06:44:15 mailstream.surrealpolitiks.com stunnel[4045]: LOG4[ui]: Service [Kick] needs authentication to prevent MITM attacks
Oct 11 06:44:15 mailstream.surrealpolitiks.com stunnel[4045]: LOG4[ui]: Service [Telegram] needs authentication to prevent MITM attacks
Oct 11 06:44:15 mailstream.surrealpolitiks.com stunnel[4045]: LOG5[ui]: Configuration successful
We might take note that it warns we should have authentication to avoid "Man in the Middle Attacks" but my searching tells me this should not obstruct my connections, and on the basis of that information I conclude my problem does not stem from this.
I have configured these services to listen locally on sequential ports beginning with 19351, and we can see from netstat that they are, in fact, listening.
root@mailstream:~# netstat -l -n |grep 1935
tcp 0 0 0.0.0.0:1935 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:19351 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:19352 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:19353 0.0.0.0:* LISTEN
root@mailstream:~#
Here is an example stunnel service conf file
root@mailstream:~# more /etc/stunnel/conf.d/entropy.conf
[Entropy]
client = yes
accept = 127.0.0.1:19353
connect = ingest.entropystream.live:443
verifychain = no
sslVersion = all
options = NO_SSLv2
root@mailstream:~#
The others are functionally identical, just change the host.
In my /etc/nginx/nginx.conf I have configured several "application" lines I use to facilitate my RTMP relays, but for our purposes we'll narrow in on the one I am using to test Stunnel.
#Enable Stunnel Test
application stunnel {
live on;
record off;
#Push, restream RTMP
#Push Telegram
push rtmp://localhost:19351/s/streamkey;
#Push Kick
push rtmp://localhost:19352/streamkey;
#Push Entropy
push rtmp://localhost:19353/live/streamkey;
#Push HLS
push rtmp://localhost/nunyabiznis/streamkey;
}
Now, interestingly, I discover something even as I write this post.
I had not used Entropy in my setting this up previously. Only Telegram and Kick, and neither worked. Both gave me different sorts of trouble. I added entropy before beginning the test that formed the basis of this documentation of the issue.
I just went and began my test stream, and Entropy is working. No errors, no problem, smooth sailing on Entropy.
So, now I have shown that Nginx and Stunnel are working with the configuration outlined above.
But, I have problems with Telegram and Kick.com still, and these problems present differently from one another.
I have just gotten a CloudFlare error attempting to mark the log output as code. Not sure what is happening. Have I made my post too long? Perhaps I should hit post and continue in a reply.
Comments
As I was saying, this is sort of odd.
Not sure what to make of that, but I'll continue...
Just as a note in advance, I/we would ask that any replies remain on-topic
I keep getting that Cloudflare message above, which is bizarre. Pardon me for following up in this way, with numerous messages. I am certainly not inviting CloudFlare to do this to me, and not trying to bump, I'd far prefer to have had this in one shot.
The most pertinent part of the error log appears to be this.
SSL_read: ../ssl/record/rec_layer_s3.c:308: error:0A000126:SSL routines::unexpected eof while reading
There are no errors from Telegram, that comes out in the stunnel log after Kick starts.
Neither Telegram nor Kick make any indication there is an effort to connect to them.
It seems I cannot mention slash var slash log slash stunnel without trigering this cloudflare thing.
Trying to post the log without mentioning the filename.
2023.10.11 07:59:58 LOG5[155]: Service [Kick] accepted connection from 127.0.0.1:39956
2023.10.11 07:59:58 LOG5[155]: s_connect: connected 52.223.227.122:443
2023.10.11 07:59:58 LOG5[155]: Service [Kick] connected remote server from 172.245.6.138:60796
2023.10.11 07:59:58 LOG3[155]: SSL_read: ../ssl/record/rec_layer_s3.c:308: error:0A000126:SSL routines::unexpected eof while reading
2023.10.11 07:59:58 LOG5[155]: Connection reset: 7348 byte(s) sent to TLS, 3447 byte(s) sent to socket
2023.10.11 08:00:04 LOG5[156]: Service [Kick] accepted connection from 127.0.0.1:39960
2023.10.11 08:00:04 LOG5[156]: s_connect: connected 52.223.227.122:443
2023.10.11 08:00:04 LOG5[156]: Service [Kick] connected remote server from 172.245.6.138:60808
2023.10.11 08:00:05 LOG3[156]: SSL_read: ../ssl/record/rec_layer_s3.c:308: error:0A000126:SSL routines::unexpected eof while reading
2023.10.11 08:00:05 LOG5[156]: Connection reset: 7093 byte(s) sent to TLS, 3447 byte(s) sent to socket
It's probably the WAF.
It happens when you don't use the code tags it seems.
Since Telegram is producing no errors, I think that I'm actually connected to Telegram but there's a problem with the stream key. I'm connected to the Telegram RTMP server, but because the stream key is getting mangled in some way, the video output does not appear.
Telegram, it is worth noting, does something odd with their stream keys which I've not seen elsewhere. They insert a colon in the middle of it.
Surely, you'll pardon me for not posting my stream key here
But it looks like this #########:#Xx#xx#xXxXxx#XxXXx#Xx
Where # is a number and X is a letter, and you see that : right in the middle.
It occurred to me that this might be causing some trouble in nginx.conf, being interpreted as a port or some other distinction.
Part of what prompts this is that I cannot stream to Telegram directly from XSplit Broadcaster either. It begins, the video is visible, and then it disconnects. Right now when I stream to Telegram, this is facilitated with restream.io, and I'd like to free up that service since it only allows two outputs free.
I do not have the same problem in OBS. Using OBS I can stream directly to Telegram, but not through the Nginx relay.
So, I tried to "escape" the colon, replacing it with %3A as instructed by this tool here.
https://httptools.dev/url-encoder-decoder
Using the same stream key in OBS, but with %3A instead of the : colon, OBS is able to stream to Telegram, which tells me that I'm escaping it correctly and at least OBS is able to interpret this that way.
This does nothing to help my problems with the relay.
I get it... I was trying to say "here is the output of the file" and then put that output in code tags.
/var/log/stunnelThat worked. Thanks!
This is about as far as I've gotten with this.
It appears I've got two problems. There's the stream key issue above, and some kind of SSL problem connecting to kick.com.
This error
SSL_read: ../ssl/record/rec_layer_s3.c:308: error:0A000126:SSL routines::unexpected eof while readingAppears to be a common one, but most of the posts I find describing it are old and suggest that I can fix it with updated software.
Given that my software is quite new compared to those posts, this does not appear to be my issue.
I'll be most grateful for any assistance.
Just a first hunch. Have a look at that file, line 308, and try to find out what causes the assumption that some file being read is incomplete - Hunch: it's the colon.
In case you can't make sense of C code, feel free to PM me say, line 300 to 320 and let me know the exact version of your linux, kernel, and SSL.
P.S. [d|l|s]trace can be a good friend in situations like this. e.g. to see what that code is trying to read (hint: 'file' carries little meaning in Unix, can be pretty much anything, e.g. a TCP stream).
It's wild to me RTMP remains a standard in a world of stable SRT, WebRTC, and soon RIST. Anyway, if you're open to a you should use X instead comment, lmk. There are better solutions for restreaming (even RTMP) nowadays.
I am open to alternative methods of getting the RTMP feeds to these providers. If you are suggesting I do something other than RTMP this will not be useful since I am in no position to tell the syndicates how they should be receiving my video feed.
As to why it remains a standard, I would say that whatever benefits there may be to the other protocols you mention, RTMP works very well. Lots of software has been designed around that protocol, and streamers who are not technically inclined are not going to be enthusiastic to change their configurations anymore than administrators are going to be eager to incur the tech support burden associated with this.
I'm the kinda guy who tries to fix things that aren't broken. Others are smarter than me, and are disinclined.
I'm more than a little unclear as to how I would do this. I've been trying to find the file in question and not having much luck with it.
I've had luck with two systems in the recent past.
The first is Datarhei Restreamer. It's a Node.js app that is very user friendly and supports just about everything you could want from a re-streaming server, even creating a nice page for you with your HLS feed streamed to a web-based player. The deploy is easy if you're okay with their Docker container.
The second system I've had luck with is SRS. This is a streaming server with configuration similar to Nginx, but with a ton more features. I like how portable the config is in this one.
The former is much more user-friendly, the latter is much more administrator friendly. The latter is probably more performant at extreme scale being written in C++. Both support a ton of protocols out of the box and both (and perhaps others) will support AV1 when it becomes widely adopted in the future while RTMP will always be stuck on H.264.
Certainly I can't make some big video streaming service support something new.
I think if you have a perfect Internet connection it can work great. But for the kinds of on-location, "IRL," streams I support, including streams over cellular networks and public WiFi, it's often a mess of pixelation and sometimes is hard to recover with a spotty connection. RTMP was a protocol designed for the Internet of the early 00s. The world is different now with better compression algorithms and good UDP video streaming protocols. Of course, if you're sitting at home or in a studio with a great wired connection, it probably won't matter too much.
These comments mostly refer to getting the video video to the re-streaming server. Between the server and the streaming service, you are at the mercy of what they demand.
I think in the IRL niche things might be a little different. But I get where you're coming from.
The error message told you: '../ssl/record/rec_layer_s3.c'.
In case that message were relative to some (whatever) directory deep within the build (which is quite probable) the 'find' command (or preferably something better like e.g. fd-find) with 'rec_layer_s3.c' as a parameter (what to find) should do the trick.
If your search fails, just tell me the exact program and version of your "stunnel" and "openssl" software.
P.S. the problem you experience seems to be known and there is a fix. It's related to your SSL version (hence my question re. the exact version). You can read about here: https://github.com/openssl/openssl/issues/18866#issuecomment-1194219601
Thanks!
It looks like I might be able to fix it from this thread by updating Nginx, which I am about to attempt.
EDIT: Strike that, it looks like if I do this it will break Webmin/Virtualmin.
You're welcome.
But still, openssl and linux distros being what they are (I want to stay polite here, hint, hint) be prepared to find out that your distro/version might not have a recent enough version (> late 2022 and later) ...
I had to back burner this and just came back to it.
I tried installing the latest nginx on a different VPS. Then tried to install the RTMP module on that, and come back with an error.
root@root:~# nginx -v
nginx version: nginx/1.25.2
root@root:~# apt install libnginx-mod-rtmp -y
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:
The following packages have unmet dependencies:
nginx : Conflicts: nginx-common but 1.18.0-6ubuntu14.4 is to be installed
E: Error, pkgProblemResolver::Resolve generated breaks, this may be caused by held packages.
root@root:~#
I'm going to go out on a limb and say it has to do with the ubuntu-repo version of Nginx vs a current nginx repo.
https://launchpad.net/~ondrej/+archive/ubuntu/nginx-mainline
sudo add-apt-repository ppa:ondrej/nginx-mainline
sudo apt update
then try running apt-get install libnginx-mod-rtmp and you should be good to go
Thank you. Tried that, and I think I comprehend the reasoning.
I had to go through a few steps to get the latest version of Nginx to install via apt. So, it makes sense that the module is expecting an older version and getting jammed up on account of this.
Feel free to correct me if I'm wrong, I'll appreciate it.
So, I took those steps and I meet a very similar error. Except that this time it is expecting the correct version of Nginx. So I checked my Ubuntu version, and it appears to be expecting a different version of Ubuntu.
Here is the output.
Do you have anything running on Nginx now?? If so, I'd backup your configs - do a apt-get purge nginx and then re-run the nginx installation and then drop your configs back in.
To me, it looks like the ubuntu-version of nginx is installed and just need to be replaced.
This is a freshly reinstalled VPS with no configurations or data set up.
I put it together for a test because I was trying to get RTMPS working on a Virtualmin server and upgrading Nginx (which seemed prudent from some documentation I read) looked like it would break Webmin.
The entire command history since reinstall is;
Note that, I'm running this as root and the times I used sudo was just as a consequence of C&P from instructions on the Nginx documentation.
Everything looks good there. I don't know if Virtualmin needs a specific vesion of nginx (seems like it'd be weird if it did) but that seems to be the only apparent variable between what you've got running and what I've got running.
I don't know how much time you have to be messing with all of this stuff - but a clean 22.04 install+that repo I had posted earlier WILL work. Maybe try installing Virtualmin after you get that installed/working properly?
Virtualmin isn't on this server. (Hence its install not in the command list)
I have an Nginx rtmp relay set up on a separate Virtualmin server. I was trying to get Stunnel set up for RTMPS, which the Nginx RTMP module does not natively support.
There were substantial problems with this, leading to the creation of this thread.
Documentation I read and earlier replies in this thread, caused me to think maybe the older Nginx version Webmin installed might be the problem.
When I tried to install the updated version of Nginx, it said it would remove Webmin, and that hardly seemed like a good idea since that server is in production.
For that reason, I went to this other VPS I bought a while back, reinstalled the OS, and issued the commands you see listed above.
Ok I think I understand the situation better now. So the separate VPS - were you able to get libnginx-mod-rtmp installed ok?
No. That's the whole point of our exchange. All of the error messages I've provided today are from this newly installed test VPS.
After stalling in my progress on October 11th, I put this project on the back burner, and returned to it today testing on the new VPS without Webmin/Virtualmin.
After receiving the errors there, I came back to this thread. All of what I have said today pertains to the clean test VPS.
Have you tried forcing apt?
apt -f install
Just tried. No dice.