New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
I might try that! Thanks. I've been kind of harsh on edis.at, and really, they've been very polite... I know they're trying.. and if this were web hosting, EDIS would be more than adequate. :-)
Would a VRS package offer networking advantages over KVM? Right now, I have the "KVM Plus" package.
Moving is a fairly complex procedure with consequences. (like cached IP numbers in shoutcast clients...)
I don't want to do it unless there's a fairly decent chance Germany + VRS will pan out. By "pan out" I mean, it's OK to go down once a month, perhaps even six times in a particularly bad month, but daily would not be acceptable.
It is difficult to get found by people who like this niche type of music, when they do find me, I don't want to drive them away. They don't see that everyone got bumped, as a result, some people take getting bumped personally. (and it messes up directory listings, who score partially on reliability)
The location so far has been fairly international, mostly from Japan, but a fair amount from Germany, Belgium, Russia, Canada, and the US. Europe is probably a pretty good choice.
There haven't been any outages over the last 8 hours, and I've attempted to load it down, to induce an error. Whatever this is, it doesn't seem to have anything to do with the number of listeners or the bandwidth that I use... it almost has to be something external.
Not yet. Now is alpha phase and beta phase starting soon (during next week).
Watching the errors today, yep, it's ALL the streams, not just the 64k. This is indeed a network issue. (and of course, it happened when I had listeners connected)
I checked with the datacenter in Sweden (Portlane):
No issues
I checked the bridge:
No issues
I checked the Switch graph:
No issues, no terminated connections
I even had them check the router:
No RST
Network load on the host 3%, PPS lower than on my home connection, Bridge load under 0,1% (since it is 10GBE)....
I have no explanation for this, and i doubt this is related to us - I can simply neither reproduce it nor watch it happen or see anything of it anywhere.
My Netcat test also runs since yesterday at roughly 1 character per second (sending unixtime) (SE->AT, AT->SE) with not a single dropped stream or package....
I can send you the errors if you like?
It's as if the network were saturated. SSH connection didn't go "DOWN" but, it was unresponsive.
ALL the streams up/down/up/down/up/down over and over. The web server (nginx) also failed. The errors were sort of flurried, there are 86 "RED" messages in my inbox from 2012-05-17, most of them really close together, between 09:48:15 - 10:43:10
(CDT, it's now 10:53, so this happened within the past hour)
Next time it happens, is there something I can do that would help get you some information?
My theory is it's related to something happening on the physical server I'm on, something is saturating the connection.
I had hoped it was something weird with my setup, but, when all the streams, ssh and the web went down, I'm pretty sure thats not it. (I haven't patched sshd :-) )
No.
I checked the switch graph, there was no load spike...
A Wire/T shark dump of eth0 would be good to have.
But for now it seems that we are out of ideas what this could be (after all you are the only one expiriencing this on the node)
Also, if you want, I'd be happy to install something you can use to watch it go down, or, you could run mplayer on one of the streams. (with -ao null so you don't have to hear it if you don't want to :-) )
That would show whats happening.
Hmm.. I've never used wireshark, (just tcpdump) would such a thing rapidly fill up the hard drive during the "non-outages" ?
I'll gladly run and install it, if you tell me what you want me to do with it... I'd do most anything to fix this problem! :-)
I should also say, these were FROM two different hosts:
My machine as well as the one providing the source stream.
I was able to access the host providing the source stream during the outages, and, I was able to access other websites. I wasn't able to access the EDIS server.
Anyway, if you tell me what to do with wireshark, I'll gladly do it.
http://wikisecure.net/linux/how-to-install-wireshark-in-linux
Just capture eth0 for a few seconds/minutes and gzip the dump.
I'm not sure if this is the right place to offer support, but if you have no problems with that, we can offer support via LET as well ;-)
Did you already check your servers' logfiles? How much RAM did you book?
What product do you exactly have right now?
Could it be the case, that you are just running out of memory?
Best regards,
Gerhard Kleewein
@William - I assume this has to happen during an outage?
Right now, everything is great, the outage is over.
You mentioned no one else is experiencing problems, maybe they are, but it's not as visible on web applications? because it's such a flurry of up/down it'd be hard to notice it unless you happened to be using ssh at the time or something.
I'll prepare wireshark and hope there's a way I can start it next time it goes down.
@ExPLOReR:
You're probably right! LOL! not the best place for support... (I sent an email earlier to you, regarding Germany, I can't find my client ID, I gave you my domain name and IP numbers. You may not have recognized me?)
I'm not using much RAM or CPU, I went through the crontab, couldn't find anything that looked suspect. The load averages just after the last outage were around 0, 0, 0 the machine wasn't loaded down. (and it shouldn't be, it's dual core, with hardly anything running) there were no runaway processes, no "fork bombs".
I can't really ID anything all that unusual/unexpected in the logs, quite a few of these (shoutcast): Socket error while waiting to send data. err=Broken pipe(32)
In icecasts error log, I can see:
Disconnecting source due to socket timeout on the source connection
And the usual, expected results of trying to fallback, Just exactly as one would expect from a network outage. (icecast/shoutcast logs are quite verbose, but it's all normal activity from what I can tell)
nginx had some network errors too.
Nothing unusual in /var/log either.
Overall, it's a very quiet, under-utilized machine.
I don't want to post my email address in public, but the email I sent you had geniegate.com in the domain name.
Are you using the most recent version of shoutcast / icecast software?
Did you already check the forums, because the error messages you posted seem to be quite common ... does it always happen with the same song? what has nginx to do in this ensemble? Do you maybe have conflicting port setups?
I also found a posting that refers to a known intro/backup problem in shoutcast and I found the same error-messages in this context ...
http://forums.winamp.com/showthread.php?p=2726491
Cheers,
Gerhard
No conflicting port setups, icecast is running on a different IP number, port 80. Shoutcast is port 5000, nginx is port 80. Usually, everything works great. So if it were a port issue, I would expect to see a lot more problems.
That shoutcast error IS very common (and unfortunately, pretty much useless in the current context), you'll see it on pretty much every network error, when the source goes down. (but I've verified, the source isn't going down)
The shoutcast server does have known problems. It's a custom build, and initially, I thought thats where the problem was too. (custom built by the shoutcast people, I don't have access to the source code) I know it's not shoutcast because icecast fails at the same time. (all three streams fail, 4 if you count shoutcast)
icecast, yes, recent. (and the master icecast server, on another machine, works fine)
It's not related to the song being played.
Last night, I added 20 bogus listeners, just to put it under heavier load than normal.... no problems.
nginx has nothing to do with it directly, neither does ssh, however, they're related to networking. If icecast, shoutcast, nginx, sshd all have network related problems at the same time, then I figure it's probably network related.
nginx is interesting in one respect, during an outage, it "mostly" works. You have to really try to see the problem. This tells me the network is going down, for perhaps a second or so, then coming back up, then going down, up, so, if other people are seeing this, they probably wouldn't notice it.
As pr. Williams request, I have tshark running, every 5 minutes it starts a new capture file, rotating around the minute. (so I should always have one hours worth of dumps, it should work, I'm running it under 'screen' so that if ssh fails, it will still run)
Hopefully this will tell us something on the next outage.
It's doing it again, but it looks like someone has rebooted it, (twice?)
VNC isn't responding either.
@gouf i'd say run ping -i 0.1 -n your.stream.source.ip from the vps and log it to a file. Later you can analyze the file and see if there was any short term packet loss during the freezing of the stream.
By the way what is the protocol that the streams run on, tcp or udp?
Protocol is tcp, (streaming MP3 is pretty much a quick-n-dirty hack off of HTTP, you can easily disect them with a perl script..)
I know there are packet losses, the thing is, if you were to poll it, you might not even see the problem. The reason I see it, is my monitoring is a persistent connection, with a constant stream of data. When the connection goes down, I know about it right away.
I confirmed they have rebooted it for some reason, hopefully they found whatever it was? :-)
Nope. reboot didn't solve the problem, must have been an unrelated thing...
I'm sending the tshark dumps to: gerhard.kleewein with the edis period at email address, hopefully they'll get to the right person.
(I looked at the dumps, couldn't really see anything, but maybe William will be able to interpret them?)
No, that was related to an attack - Not to you.
Your dumps are fine, nothing to see in them - We are out of ideas.....
Why not go with a stream provider instead? http://egihosting.com/steaming-media/shoutcast-streaming
I've found a good one (I hope) :-)
Uptime on the new host so far, seems pretty good. Since that time, I've seen, perhaps 5 outages at edis.at, (in Sweden), so, I know it's not my software. I'll probably end up switching.
edis would probably be adequate for hosting non-critical stuff, and their staff did try to help. They're not "all bad" :-) (and some people do give them high remarks, I have to think those are other locations, I probably just ended up on a bad server.)
I do hope they can find out whats wrong with their network, it's probably NOT an easy task because it's so intermittent. VERY hard to fix something when it starts working again 4 minutes into the investigation. :-/
Can you PM me the name of the provider?
HI! It's just strange, that what you call "network problems" occurs in all remote locations. These are not "our" networks - we are talking about >the internet< and different internet upstreams from major global carriers in different, globally independent locations ;-)
Anyway. We tried to figure out what your described network problems could be caused by. We are officially closing this case now, because none of our carriers were able to trace any network related problems.
We all hope you'll find the right hosting company for your services.
All the best,
Gerhard & the team