New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Any insider info on iwstack.com current downtime?
Hi,
I know this is not a support forum, but as far as I know they represent themselves here.
Is anyone know when they will be come back?
Any insider? 
They says:
Update (15:03 CET): We see everything up so it should be no reason your VM is still down, but if it is, please try a manual restart. something might have gone wrong with your ACPI at shutdown and it might have gone down unorderly.
source: https://prometeus.net/billing/announcements/48/IWStack-move-to-the-new-datanceter.html
However their OWN website is still down. ( iwstack.com )
Comments
also all my VMs on milano/dc2 are offline (but in running state).
i'm pretty tired of this whole cloudstack stuff, worked well for some years but had more than one problem on last months (impossible snapshots, stuck VMs).
did not found any useful info online or on their support.
Tag @Maounique
@alepore If you login at prometeus.net you can even view your server in console, but cannot setup the networking...
... SOMEONE WHO ACKNOWLEDGES THAT IT ISNT A SUPPORT FORUM!!! OMG CRAP BATMAN!!!!!!!
...but yet they're asking for information that if they knew, they'd probably have announced somewhere. We can't win every day I guess.
Probably something related to the fact they are migrating to a new DC (today)
Better than nothing!!
Nvm, I have to read more than the title only.
Crap, I hate days like this!!
Same story, every time. Everything will be taken care of within a couple of hours ... and then a couple of hours passes ... and a few more hours ... and a few more ... service still down... and no update or or answers to tickets.
For God's sake! Is it so hard to plan the migration better and do everything over the weekend?
Welcome back. Looking over your profile, at least you're only able to join our community whenever there's a problem with something-Prometeus.
Doing a migration during the week IS the better method if your provider cares about you. There's far more datacenter support personnel available during the week; most datacenters, even the fancy ones, are ghost towns on weekends. They're staffed at a level to handle the most common issues; they're not properly staffed on weekends to be able to apply the proper attention to one-off migration.
That was my view of cloudstack pretty much, so fragile and easy to break, also it seemed more unreliable that standalone VPS nodes.
time to move to OnAPP. 4,3 is rock solid, @OnApp_Terry what do you think?
Should atleast have more than 1 person replying to tickets on a weekday. We have 1 instance up, 1 stuck on starting. Downtime was supposed to be 4-5 hours. Its 11 hours and counting already.
Cloud does not mean 100% uptime,I am very sure they are working hard to fix the issues.
yap
One of my instances didn't restart after the move. Had to boot it myself. Console wasn't working and it took a while for it to be reachable, so I'm assuming it was doing an fsck. Seems to be working now but for IPv6.
Thank you.
It is true that I am not very active on the forums. Perhaps because of the language barrier and perhaps because I am that type of person that prefer to read and learn on my own. I'm not a crybaby, in all these years my support tickets can be counted on the fingers of one hand... And besides, I have my own business to run.
By the way, I found out about LET forum looking for information about Prometeus downtime last year when I could not get any response or info from their support.
This may be true, but in my opinion the hosting provider has only one primary task for their clients and this is to ensure the availability of services as much as possible to the end user. Most of my customers would not notice any service interruption over the weekend ... but today I am on the line of fire - in the front row!
Good news!
All my instances are looking up again. I hope for others are also.
Update (18:44): After some weird errors and some hosts not visible, we detected some cabling errors or nodes ending up in the wrong racks. Everything is rechecked now through kilometers of cables.
Yeah, what can I say, we screwed up, and, frankly, I kinda felt it will happen. This was a train wreck in slow motion, way too many things to do in very little time, it was bound to break some place.
Nevertheless, we did all we could to prepare:
1. Move the core infrastructure first (solus, iwstack master, some storage, some switches, even racks, create a link, test it, put it in production, deploy VLANs, duplicate many things no matter the cost as we will expand later anyway);
2. Move smaller services in small batches of only tens of servers for a week before to see how long it takes per node (with mixed results, though, but we learnt from it);
3. Make sure we have the people on site and even postpone it when unexpected things happened.
With all that, the goal was too optimistic, keeping downtime down to a few hours with no major screwups in a new datacenter with some equipment many people didnt see before, it would have been a miracle to say the least. you cannot practice with tons of equipment and many people before, only the real thing will give the measure of the task.
Also, from the very beginning IWStack was not designed with mobility in mind, from the heavy storage to the FC fabric with links to multiple racks, redundancies and also links tot he secondary storages, infrastructure also spread across 2 datacenters, but, what is worse, is that we had to grow faster than we planned. We currently are over capacity in Dallas and there is a lot of pressure to create more, in Milano we managed more or less, but at the expense of sane cabling and design.
This new DC presented the opportunity to finally get all together and redeploy sanely in racks, not like before, filling holes whenever one appeared, but putting all together in a place from the start. We also had to upgrade the network to a higher capacity to allow for faster backups of a huge and increasing quantity of data, and we thought that, since we have to move anyway, lets do the upgrade in the same time. While one big downtime is better than a few smaller ones as people only have to apply the contingency plans one, this downtime proved to be way bigger than anticipated, even by the pessimistic me.
We are still not back 100%, unrelated to the move, 2 servers previously moved broke down tonight, adding to the pressure. Salvatore lost 12 Kg these few weeks, which is not a bad thing overall, but gives the measure of the work he had to put up with. I always maintained we cannot aim for unlimited expansion, we must learn to say no more often if we are to maintain sanity in all fields.
I am sorry for this situation, what can I say, we can't even claim we learnt something because I am sure the next move, if there will be any, will take place in completely different circumstances. The bright spots are:
1. Less chaotic infrastructure in one place;
2. Modern and new DC which aims to host the elite of the industry.