Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


IWStack down?
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

IWStack down?

Hello folks,

Pretty surprise to see my first downtime on my IWStack VPS in years and apparently both iwstack.com and prometeus.net are down right now?

Anyone knows what's going on ?

Thanks,

«1

Comments

  • friendhostingfriendhosting Member, Patron Provider

    This iwstack.com site is available.

    Have you tried calling or writing to them?

  • I'm referring to their panel: master.iwstack.com

    The problem is their support is on prometeus.net which is down too

  • deankdeank Member, Troll

    Are hosts now deadpooling even before BF deals start rolling in?

    Pre-deadpooling.

  • Maybe they felt badly after being asked about their site/ssl here: https://lowendtalk.com/discussion/comment/3293872/#Comment_3293872 and now do some maintenance to catch up to 2021...

  • As well remind @Maounique was a owner of this host. Ask him.

  • Have you any news? I am worried very long outage after many years at 100%

    Thanked by 1Maounique
  • @jenkki said:
    As well remind @Maounique was a owner of this host. Ask him.

    Uncle Sal @prometeus is the owner, at least used to be

  • MaouniqueMaounique Host Rep, Veteran
    edited October 2021

    We are working to restore services, more details later. A part of iwstack is still down, rest is up for an hour.
    EDIT: I am contracted to work for Prometeus/Iperweb through my company, I am not the owner.

  • MaouniqueMaounique Host Rep, Veteran
    edited October 2021

    Ah, forgot to say, the cause is a network error, all 4 core switches went amok for some unknown reason, we have complicated VLAN setups for IWStack so the restore will take a bit longer. but it is worked on, configurations are restores in order.
    The cause of all 4 core switches failing and going full split brain suddenly will be investigated later but it is worrying. I does not appear to be a hardware issue.

  • NeoonNeoon Community Contributor, Veteran

    @OhJohn said:
    Maybe they felt badly after being asked about their site/ssl here: https://lowendtalk.com/discussion/comment/3293872/#Comment_3293872 and now do some maintenance to catch up to 2021...

    What a bullshit, you can reload a webserver to deploy ssl certs.

  • @Neoon the emphasis is one "to catch up to 2021", not about loading a ssl certificate.

    Please look at the website. Last technical announcement in 2017, quoted isos five years old or older, etc.
    When you don't know the company and just visit their websites (iwstack or prometeus) you get the impression of a neglected or unmaintained business (that is - in the best case - running on a good autopilot).

    That's why I never signed up there for a service. And nothing changed about that in the last couple of years.

  • NeoonNeoon Community Contributor, Veteran

    @OhJohn said:
    @Neoon the emphasis is one "to catch up to 2021", not about loading a ssl certificate.

    Please look at the website. Last technical announcement in 2017, quoted isos five years old or older, etc.
    When you don't know the company and just visit their websites (iwstack or prometeus) you get the impression of a neglected or unmaintained business (that is - in the best case - running on a good autopilot).

    That's why I never signed up there for a service. And nothing changed about that in the last couple of years.

    It does not look dated too me, besides.

    Most provider websites on LET are dated, including the OS selection.
    Not to speak of the fucking Looking Glass pages that don't even got a bloody TLS cert.

  • MaouniqueMaounique Host Rep, Veteran
    edited October 2021

    @OhJohn said: That's why I never signed up there for a service.

    Prometeus was always B2B oriented, this is why we keep such a clean network, we went in the popular market in about 2012 IIRC with only some 10% of the infrastructure and 5% of the revenue.
    We left this market after threats of hacking and bans, it is not worth the risk to the main business model, so the popularization articles and such are no longer needed, businesses have IT department, don't need hand holding and tutorials for the most part.
    IWStack offers full access to the API, has many advanced features, it is not really for the hobbyists, people use own ISOs, own templates, the VMWare part is only open to businesses with a contract.
    We are not targeting this market, I am here only in personal capacity but I can offer info for the people which are still with us from the times we were active here until I am banned again.

    @OhJohn said: Maybe they felt badly after being asked about their site/ssl here: https://lowendtalk.com/discussion/comment/3293872/#Comment_3293872 and now do some maintenance to catch up to 2021...

    Hi, alex, I have already addressed that issue.

  • MaouniqueMaounique Host Rep, Veteran

    @Neoon said: Not to speak of the fucking Looking Glass pages that don't even got a bloody TLS cert.

    As noted before, there is nothing secret or a login needed to use the looking glass.
    Why would we need to encrypt something public, put up there for being seen by everyone?
    Some people like introducing SPoFs, overhead and lower responsiveness, I only do what is actually needed.

  • MaouniqueMaounique Host Rep, Veteran

    UPDATE: 00:14

    Everything should be working, some services might need restart.

    In particular, isolated network will definitely need a restart, please shutdown all instances with a virtual router and restart it with a clean-up option just in case.

  • MaouniqueMaounique Host Rep, Veteran

    @mgiammarco said: Have you any news? I am worried very long outage after many years at 100%

    Thank you, to be honest I am also worried until we know the cause of this issue. It might be a software issue, some update, but those were pretty far back...

  • OhJohnOhJohn Member
    edited October 2021

    See, so far I only saw prometeus from the outside (e.g. by looking at the website 2 years ago and then again when it came up in the other thread), today I got another glimpse:

    • customers reporting an outage but also stating that they never had one before for years.

    • and seeing a team that is actively and fastly mitigating the outage and reporting about it

    So while this might be a thread about an outage, it's actually two positive signs.

    And if your website or the issues raised about it is not important for your business, so be it, that is fine as well.

    Thanked by 1Maounique
  • NeoonNeoon Community Contributor, Veteran

    @Maounique said:

    @Neoon said: Not to speak of the fucking Looking Glass pages that don't even got a bloody TLS cert.

    As noted before, there is nothing secret or a login needed to use the looking glass.
    Why would we need to encrypt something public, put up there for being seen by everyone?
    Some people like introducing SPoFs, overhead and lower responsiveness, I only do what is actually needed.

    I know but @OhJohn dosen't seem to know.

    Thanked by 1Maounique
  • MaouniqueMaounique Host Rep, Veteran
    edited October 2021

    @OhJohn said: customers reporting an outage but also stating that they never had one before for years.

    There have been outages in some services every year, just recently (some 2 months) one of our OVerZold nodes failed and we are still running on a temporary machine until someone can go there and solve the problem. That has taken longer than anyone thought due to illness and corona issues, the VMs are working but the panel reports errors so some operations we need to do manually on the node for the customers. In the end we are moving out the remaining containers, some 12 are left there.

    IWStack as a whole had failures here and there, especially the SSD local storage nodes fail from time to time and live migration is not available for them due to obvious reasons, so the outages tend to be hours long, but, fortunately, that is about one node a year, so, last 2 years or so we have better uptime than Azure and AWS, for example (those that I have checked).
    Of course, this issue set us back a considerable distance, but I still think we are at least tied with AWS for the year.

    Of course, our prices are higher than at least half of the hosts here, but we also offer higher uptime and features. I don't know any product here which can compare with IWStack regarding features (real isolated networks, own ISO, own templates-created here or uploaded-, ful access to CloudStack API, load balancing, external firewall access, charge by the hour (actually 10 minutes...) it is a very complex product with high maintenance we are only able to offer at these prices because we have it up for our corporate customers too.
    https://www.iwstack.com/price-calculator.php Some instances can meet the LEB price requirements due to this kind of dual use.

  • I used @prometeus from 2015 to 2020. I never faced any major issues. Support was quick and very friendly.

    Thanked by 2Maounique desperand
  • @vedran said: Uncle Sal @prometeus is the owner, at least used to be

    Not exactly.

  • MaouniqueMaounique Host Rep, Veteran
    edited October 2021

    UPDATE

    We are getting many tickets about not working instances. Due to the network issue, some have remained stuck.

    Please follow the previous advice: Stop the instance and start it. Restart might work but it is better to follow the full "cold" boot procedure.

    Also, please remember the virtual router issue. The virtual routers we affected by the network issue, please restart with the cleanup option. If it is still not working, you will need to shutdown all VMs behind the virtual router, even if they only have one interface linked to it.

    As noted, ATM everything about the infrastructure is looking green, but the incident confused some VMs which had secondary storage loaded (ISO linked, mainly) as those are loaded over NFS, not the SAN fabric.
    Please try to avoid leaving ISOs attached when not needed, either in regular VPSes as well as in IWStack instances. Not only in case of network issues that they will probably crash the VM, but we are sometimes updating the ISOs and this will also crash your VM in most cases.

    It looks like a server in the rack rebooted at the same time so we are suspecting a power issue.

    Two of the switches were in that rack and were briefly off and the other two thought were alone and took over the duties of the down ones which did not acknowledge it probably because the downtime was very brief.

    We will re-check the ATS in the rack.

  • It's solved on my side.

    Sidenote : It would be nice if iwstack / prometeus had a real "status" or "outage" page because yesterday I couldn't find any trace that they were aware of the problem and working on it, thus my post here.

    Anyway I'm still very happy with them, 99.9684% since I installed hetrixtool on 20 May 2019 so I guess yesterday's was the only downtime since then.

    Thanked by 1desperand
  • MaouniqueMaounique Host Rep, Veteran
    edited October 2021

    We still have issues, unfortunately. there will be more updates as they come.
    At this time the SAN storage has problems, nothing we can solve from here, we do not know what is going on, the DC is trying to fix it.
    We suspect some power issues at the DC, the only common thing that could bring these unrelated problems at the same time.
    I am really sorry, we are doing what we can without being present there, which is pretty hard.

  • @benjamin74 said:
    It's solved on my side.

    Sidenote : It would be nice if iwstack / prometeus had a real "status" or "outage" page because yesterday I couldn't find any trace that they were aware of the problem and working on it, thus my post here.

    Anyway I'm still very happy with them, 99.9684% since I installed hetrixtool on 20 May 2019 so I guess yesterday's was the only downtime since then.

    It was mentioned in the prometeus's announcement page, although that is down now too.

  • MaouniqueMaounique Host Rep, Veteran
    edited October 2021

    Yes, something really bad happened at the milano datacenter, everything is down more or less.
    We do not have access to check and have to rely on technicians there which are un familiar with our SAN which costed 1 million and is very sophisticated.
    Salvatore is in hospital too so it is a very bad situation.
    The DC was very reliable for many years. apart from one power issue and one fibercut which happened when they were moving to the new building and everything was split more or less. Now they deny there is a power issue but I can't think of anything else that could affect so many things at the same time, things which are unrelated. For example, the SAN does not use the network has it's own fabric, should have not been affected by a network issue.
    After the problem of 2 days ago we asked them to switch our switches into another power line and they are up, the SAN, while in another location (we thought it might be our ATS 2 days ago so didn't think the SAN could be affected and it has UPSes and power regulation built in) it is affected too.

    Thanked by 1wcypierre
  • MaouniqueMaounique Host Rep, Veteran
    edited October 2021

    cdlan site is down too. We are located here:
    MIX in Caldera 21
    DC CDLan – Building Yellow/Piastra – Floor A
    Caldera21 Datacentre
    I think whatever happened to us is now generalizing in the whole building.
    Why all bad things have to happen in the weekend...

  • MaouniqueMaounique Host Rep, Veteran

    CDLan site back up. I have some hope.

    Thanked by 1wcypierre
  • MaouniqueMaounique Host Rep, Veteran

    The Hitachi technicians are working on SAN, they are trying to figure out how to recover one file system, 2 others are up.

  • Any news? Here my customers are open they must work I need to decide if I have to restart my backups in another provider

Sign In or Register to comment.