Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Sign In Register

Quick Links


BMail.ag - Secure Email Service
Server.net
CPLicense.net
VPS Server
Buy VPN
Vultr
VMs for AI
HostDare
ReliableSite White-Label Dedicated Hosting for Resellers
25% Recurring Discount on NVMe VPS
Try EnsoVPN - Reliable VPN - 1-Day Free Trial
K.N Cloud — High-Performance KVM VPS in Miami,Frankfurt and Amsterdam
InterServer VPS
BMail.ag - Secure Email Service
Best VPN
High-Performance Bare Metal Server Solutions
Karvl.com
Server Mania Cloud Hosting
DataWagon Hosting
AlphaVPS Hosting
Evoxt.com
Clouvider
VPS Hosting with NVMe
Residential IPs in the US & 4G Mobile Proxies in EU & US with Unlimited Bandwidth
ReliableSite White-Label Dedicated Hosting for Resellers
Rabisu - Hosting Solutions
CloudLinux
Try EnsoVPN - Fast & Private VPN - 1-Day Free Trial

Categories

In this Discussion

Home › Providers › Outages
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

1GServers outage?

1235»

Comments

  • OhJohnOhJohn Member

    Rebuilding degraded Raids right now, happy that I shutdown the server immediately after I got the temperature warnings. Disks seem to have survived.

    Thanked by 11gservers
  • thanethane Member

    @fxxxxknvwinmac said: cost me nearly $2000

    So far... Fallout will extend over the next week for affected companies. Still no word about what is taking so long for 1g full recovery, very concerning. I understand it's stressful, it's stressful for everyone, but we should deserve at least a description of what's happening so we can make plans.

  • fxxxxknvwinmacfxxxxknvwinmac Member

    @thane said:

    @fxxxxknvwinmac said: cost me nearly $2000

    So far... Fallout will extend over the next week for affected companies. Still no word about what is taking so long for 1g full recovery, very concerning. I understand it's stressful, it's stressful for everyone, but we should deserve at least a description of what's happening so we can make plans.

    Yes, I have tried calling and emailing, but received no response at all. It’s really quite frustrating. Please excuse my rude comments due to my anxiety.😭

  • OhJohnOhJohn Member

    @thane: as soon as I read this by PhoenixNAP (recovered): https://status.phoenixnap.com/incidents/dnjp5pfv2mmh I logged into the 1gservers portal, repowered the server via the reboot button there, checked temperatures (fine now) and than started rebuilding the Raids.

    If your server is still offline, I would try the repowering button and see if you may have a good ending for the day.

    Thanked by 1thane
  • thanethane Member

    @OhJohn thanks, but no power button in my interface, nothing populating in the device controls either.

    @fxxxxknvwinmac right with you brother, long morning, long day, long night. Will it be long overnight as well? I really hope not :/

  • szarkaszarka Member

    Praise the Lord! My server is reachable again and things are looking good. /me crosses fingers

    Thanked by 11gservers
  • smrgarbsmrgarb Member

    @szarka said:
    Praise the Lord! My server is reachable again and things are looking good. /me crosses fingers

    Far out that's crazy, I wonder if it was a specific part of the DC that failed and they switched off all servers to be safe.

  • 1gservers1gservers Member, Patron Provider

    @houjiangen1 said:
    I ordered a server for $54, but now they’ve sent me an invoice for $89. Have you ever encountered this situation?

    Send an email to [email protected] and we will fix that.

  • 1gservers1gservers Member, Patron Provider

    @aru said:
    they should have a status website

    That’s a great idea. We’ll get one going soon!

    Thanked by 1szarka
  • 1gservers1gservers Member, Patron Provider

    @OhJohn said:
    ok, at least i seem to have the servers still in the old colo location and not the new DC.

    And here is the status page by phoenixnap: https://status.phoenixnap.com/

    Incident: https://status.phoenixnap.com/incidents/dnjp5pfv2mmh

    Want to clarify that this incident was at PNAP (now RadiusDC) and not at our new center. We actually have redundant cooling (even though apparently PNAP doesn’t).

  • 1gservers1gservers Member, Patron Provider

    @concept said:
    Also I believe most of 1Gservers are being hosted in their own datacenter. They used to be solely in PhoenixNAP. I think they still have some private cages there but most stuff got moved.

    We are working on migrating key systems to our new data center. It’s a project to say the least.

  • 1gservers1gservers Member, Patron Provider

    @fxxxxknvwinmac said:
    我真的是要气死了，孩子们，这种情况出现的次数会很多吗？我准备换别家了，有什么推荐吗？

    This has never happened before with us. Cooling is restored at PNAP. If you are still experiencing issues please email [email protected]

  • 1gservers1gservers Member, Patron Provider

    @thane said:
    The outage is really something, and even though it may technically be Phoenixnap at fault, the last year has been several other downtime events from 1g network or heating issues. I'm losing confidence in them, which sucks because the first year or two was so good, and their support is generally fast and helpful. But I don't want to reach out to their support ever, I just want the gear and network to work. Now here's another big downtime event, zero communications about it, zero updates, even phoenixnap has a lame excuse for a status update. Just bad all around.

    Totally understand. I do want to emphasize all the issues have been with PNAP. This is why we built our own DC. Please email [email protected] and inquire about relocation to our new data center, which actually has redundant cooling and power!

  • 1gservers1gservers Member, Patron Provider

    @concept said:
    The only provider that is in Phoenix area and not in PhoenixNAP is GTHost, they are in Omnis Network DC at 1005 W. Geneva Dr. Tempe, AZ, 85282.

    Some may need to look at options in DFW area in TX or Los Angeles for a redundant server

    Omnis shutdown operations last year. We built our own facility to migrate to since all the issues at PNAP.

  • 1gservers1gservers Member, Patron Provider

    @iohn said:
    can confirm portal is back online but it is quite slow at the moment

    this is an improvement but i do see that my servers remain offline which explains why i can't ssh into it

    anyways im sure the folks at 1gservers are working hard to rectify the situation.

    btw the lessons from OVH Strasbourg still holds, always back your stuff up

    Just pulled 18 hours straight getting everything back online. Please email [email protected] if any outstanding issues.

    Thanked by 1allthemtings
  • 1gservers1gservers Member, Patron Provider

    @iohn said:

    @1gservers said:
    We are working on bringing stuff online. PNAP is mitigating the issue on their end. We are mitigating optics that have failed that need replacement.

    Thanks for the hard work!

    Thank you did the kind words. Just pulled an 18 hour shift to get everything online and replace failed network components.

    Thanked by 1iohn
  • 1gservers1gservers Member, Patron Provider

    @bemars said:

    @LBF said:
    Sometimes things like this happen, unfortunately. Look at the OVH fire many years ago, everything gone. Be glad it's just a cooling problem. Could have been way worse.

    I've known a guy who had servers for years at 1Gservers, never had a problem. And this one time they've got an outage almost everyone who got a server there is getting cray cray lmao. Be patient, this can happen at any datacenter/provider.

    Sorry but I just had to say this. We've all been very lucky with providers with barely any downtime.

    Yes, this can happen at any DC, but the issue is with how the outage was handled by 1GServers. Very little transparency as to what the plan is on recovery, damage done etc. For many of us that run our business of 1GServers, the lack of communication is the problem. I for one will be finding alternate providers due to the silence.

    This isn't personal, it's business -- and it shouldn't be handled this way.

    I understand your concern and agree about the lack of communication. We learn from things like this on how we can do better. We are implementing a plan to have a status site hosted off site based upon feedback in this thread.

    I just pulled an 18 hour shift getting everything back online, and we have overnight staff working on any outstanding issues. It’s definitely our priority to get you online.

    I will send a DM, hopefully we can have a conversation before making your final decision.

  • 1gservers1gservers Member, Patron Provider

    @fxxxxknvwinmac said:

    @1gservers said:
    We are working on bringing stuff online. PNAP is mitigating the issue on their end. We are mitigating optics that have failed that need replacement.

    Thank you for the fix, but it was very painful. Sir, this downtime has cost me nearly $2000, which is very distressing.😭😭😭

    Please send me a DM with your client ID or email. I will review with management.

  • houjiangen1houjiangen1 Member

    @Mainfrezzer said: 一页上的信息很滑稽。

    @1gservers said:

    @houjiangen1 said:
    I ordered a server for $54, but now they’ve sent me an invoice for $89. Have you ever encountered this situation?

    Send an email to [email protected] and we will fix that.

    I emailed [email protected] 8 hours ago but haven’t received a response. My invoice is due in 2 days, so I hope you can resolve my issue as soon as possible.

  • 1gservers1gservers Member, Patron Provider

    @houjiangen1 said:

    @Mainfrezzer said: 一页上的信息很滑稽。

    @1gservers said:

    @houjiangen1 said:
    I ordered a server for $54, but now they’ve sent me an invoice for $89. Have you ever encountered this situation?

    Send an email to [email protected] and we will fix that.

    I emailed [email protected] 8 hours ago but haven’t received a response. My invoice is due in 2 days, so I hope you can resolve my issue as soon as possible.

    I would anticipate a response within 12 hours

  • tauseeftauseef Member

    @bemars said:

    @LBF said:
    Sometimes things like this happen, unfortunately. Look at the OVH fire many years ago, everything gone. Be glad it's just a cooling problem. Could have been way worse.

    I've known a guy who had servers for years at 1Gservers, never had a problem. And this one time they've got an outage almost everyone who got a server there is getting cray cray lmao. Be patient, this can happen at any datacenter/provider.

    Sorry but I just had to say this. We've all been very lucky with providers with barely any downtime.

    Yes, this can happen at any DC, but the issue is with how the outage was handled by 1GServers. Very little transparency as to what the plan is on recovery, damage done etc. For many of us that run our business of 1GServers, the lack of communication is the problem. I for one will be finding alternate providers due to the silence.

    This isn't personal, it's business -- and it shouldn't be handled this way.

    I agree, this same situation has happened to me as well. Most of the time, they only respond after the issue has already been addressed or fixed, leaving us clueless about what is happening. Their poor communication not only frustrates us but also causes our customers to lose confidence in us.

  • windytime90windytime90 Member

    @bemars said:

    @LBF said:
    Sometimes things like this happen, unfortunately. Look at the OVH fire many years ago, everything gone. Be glad it's just a cooling problem. Could have been way worse.

    I've known a guy who had servers for years at 1Gservers, never had a problem. And this one time they've got an outage almost everyone who got a server there is getting cray cray lmao. Be patient, this can happen at any datacenter/provider.

    Sorry but I just had to say this. We've all been very lucky with providers with barely any downtime.

    Yes, this can happen at any DC, but the issue is with how the outage was handled by 1GServers. Very little transparency as to what the plan is on recovery, damage done etc. For many of us that run our business of 1GServers, the lack of communication is the problem. I for one will be finding alternate providers due to the silence.

    This isn't personal, it's business -- and it shouldn't be handled this way.

    Partly agree. But when their web management was down, it's hard to notify their customer over their email system. It would be more helpful if he posted some updates on this forum sooner.

    I have survived a lot of incident like this, including OVH, NOCIX, Quadranet and 1G is far better then the other. I am seeing that they are trying their best to bring everything online, even help me with technical stuff like pulling out, pulling in drives, changing servers because of NIC failing... I think they are doing their best they can, so a little of lacking communication is totally understandable.

  • thanethane Member

    To me it wasn't even the "lack of communicating", it's the info being passed to us. I get a reply to tickets very fast, but I get "we see the matter. We are working on the matter", it's a big middle finger to me. It would have been better to hear a canned "network is borked, we're troubleshooting to figure it out". I got better details of what was going on by watching the reports from other providers, which is not a good look. I'm looking at namecheap/liquidweb/pnap status pages and social media to get clear information, while my provider tells me "we're working on it", what is "it"? I need something to pass on to my customers, I need something that tells me whether we can wait longer or need to implement DR.

    I'm very interested to learn more about 1G's new DC, because it does seem most issues are a result of PNAP issues. How did PNAP not have redundant cooling in place? One thing is for certain, PNAP is toast, although giants like Namecheap with 5k servers there are probably stuck.

  • bemarsbemars Member
    edited 4:42PM

    @windytime90 said:

    @bemars said:

    @LBF said:
    Sometimes things like this happen, unfortunately. Look at the OVH fire many years ago, everything gone. Be glad it's just a cooling problem. Could have been way worse.

    I've known a guy who had servers for years at 1Gservers, never had a problem. And this one time they've got an outage almost everyone who got a server there is getting cray cray lmao. Be patient, this can happen at any datacenter/provider.

    Sorry but I just had to say this. We've all been very lucky with providers with barely any downtime.

    Yes, this can happen at any DC, but the issue is with how the outage was handled by 1GServers. Very little transparency as to what the plan is on recovery, damage done etc. For many of us that run our business of 1GServers, the lack of communication is the problem. I for one will be finding alternate providers due to the silence.

    This isn't personal, it's business -- and it shouldn't be handled this way.

    Partly agree. But when their web management was down, it's hard to notify their customer over their email system. It would be more helpful if he posted some updates on this forum sooner.

    I have survived a lot of incident like this, including OVH, NOCIX, Quadranet and 1G is far better then the other. I am seeing that they are trying their best to bring everything online, even help me with technical stuff like pulling out, pulling in drives, changing servers because of NIC failing... I think they are doing their best they can, so a little of lacking communication is totally understandable.

    This forum here is not a replacement for a proper incident reporting platform. There are now a plethora of tools that do just that (eg: Statuspage from Atlassian is purpose built for it). I have a dozen or so other vendors that all report through that platform (or similar) and everything is posted through Slack/Email so that our team can manage our own downstream clients.

  • conceptconcept Member
    edited 4:45PM

    @thane said:

    I'm very interested to learn more about 1G's new DC, because it does seem most issues are a result of PNAP issues. How did PNAP not have redundant cooling in place? One thing is for certain, PNAP is toast, although giants like Namecheap with 5k servers there are probably stuck.

    This is the most we get in terms of information on their new DC.
    https://1gservers.com/facility.html

    PhoenixNAP is definitely toast and there really isn't any alternatives in the Phoenix area. It doesn't help that everyone in PHX area connects through there. Namecheap CEO claims they have other datacenters but doesn't really seem like it as all their services was broken once PNAP went down.

    There isn't that many DC in PHX in general.
    https://www.peeringdb.com/search?q=PHX

    So better to look at other regions with more availability like DFW or Houston, Texas or Los Angeles. There are providers who would happily take your business.

  • bemarsbemars Member

    @thane said:
    To me it wasn't even the "lack of communicating", it's the info being passed to us. I get a reply to tickets very fast, but I get "we see the matter. We are working on the matter", it's a big middle finger to me. It would have been better to hear a canned "network is borked, we're troubleshooting to figure it out". I got better details of what was going on by watching the reports from other providers, which is not a good look. I'm looking at namecheap/liquidweb/pnap status pages and social media to get clear information, while my provider tells me "we're working on it", what is "it"? I need something to pass on to my customers, I need something that tells me whether we can wait longer or need to implement DR.

    I'm very interested to learn more about 1G's new DC, because it does seem most issues are a result of PNAP issues. How did PNAP not have redundant cooling in place? One thing is for certain, PNAP is toast, although giants like Namecheap with 5k servers there are probably stuck.

    100% agree. The details matter. I would say that the 1GServers customer base are fairly capable and experienced (this isn't Squarespace!), so give us the situation in detail. With that, I can manage expectations downstream better across our team.

    Thanked by 2thane szarka
  • conceptconcept Member
    edited 4:51PM

    Yeah there are a lot of lessons to be learned here and it seems @1Gservers gets it. I'm glad they are at least responding to all the messages here, good or bad now that it isn't all hands on deck.

    Thanked by 2oloke thane
  • szarkaszarka Member

    @smrgarb said:

    @szarka said:
    Praise the Lord! My server is reachable again and things are looking good. /me crosses fingers

    Far out that's crazy, I wonder if it was a specific part of the DC that failed and they switched off all servers to be safe.

    Well, my server itself was never shut down. Just had zero connectivity.

    Thanked by 1thane
Sign In or Register to comment.