crunchbits outage?

FAT32

@crunchbits said:

Let's do some nice deals for Black Friday? Anything (hosting related) at any price.

seoky

@FAT32 said:

@crunchbits said:

Let's do some nice deals for Black Friday? Anything (hosting related) at any price.

you mean 10C10G1T@1$/y right?

rcy026

@crunchbits said:

[lots of relevant text snipped]

I'm truly sorry to our customers for the downtime, hassle, and failures on my part and over this weekend we'll be implementing some large changes (starting internally) to alleviate this to the best of our ability from lessons learned the hard way this morning. Eligible services (meaning: not you with a yearly) are also getting a bump in SLA credits for this event.

I just have to point out that this is the way a provider should handle outages. No bullshit, own the mistakes, explain what went wrong, apologize and make sure it doesn't happen again. Well done!

ServerBachelor

@crunchbits said:

@FAT32 said:
My brain: Don't say it, don't say it
Me: The end is...

Too soon, junior

@Allay said:
Reason I think they may be shutting down is that everything's been out of stock for a while, their sales hasn't replied to me in over a month either

Stuff is out of stock because there are big changes (and I hope: well received specials) coming and I don't want people buying something only to be (rightfully) upset 2-4 weeks later that X or Y is available and they just bought Z. I also don't want my team doing a bunch of extra work to service change requests on week old deployments. Toss in a surge of existing customer custom orders/growth and my personal preference to service and take care of the ones who helped us grow before onboarding new customers. I wish I could do both simultaneously, but I just can't reasonably meet those needs yet. Additionally, after getting sick pulling ~1hr per night sleep for too many days straight I realized we (myself and my entire team) have been pushing too hard for too long and need to have a plan to reasonably continue operations being mindful of mental and physical health. Luckily this is pretty easily achievable and we already have some internal task items to deal with it, but just to shed some light on why new sales/onboarding have been taking a backseat.

Plus, I don't like not being able to interact with friends and customers (discord, LET, etc) more often in an 'unofficial capacity'. It gets too cold and corporate, and I do value a lot of the input we receive from regular chats about what everyone is doing with their hardware. You wouldn't believe how often someone says they're doing XYZ and then it clicks for us that we have a certain type of hardware that isn't a standard product but we could deploy for their use-case and save them money every month. In my opinion, that kind of attention to detail is one of the biggest reasons to go with a small provider like you'll find on LET.

Also: I think we quoted you like 2 or 3 build requests already, unless I'm mixing you up with someone else (very possible, there are email threads that look to be the same person but might not be).

@suyadi92 said:

Crunchbits website are also down

Unfortunately, we thought we removed all dependencies on third party stuff not loading breaking the website a few months ago. Either a commit was mistakenly reverted or we missed something--but our blog going down caused the whole website (which is hosted offsite and redundant in an effort to keep website/billing/discord/ip phones all separately available avenues to reach us) to 500 out. Embarrassing, honestly. Completely my fault as well, as we have these little admin things to sniff out, sort out, test properly, and fix up but I have not been giving everyone enough time to really follow through on those tasks and do follow-ups.

@haste0675 said:
so much for HA

HA as far as services you buy from us? None are, none were ever sold to you as HA. That is something we've been investigating offering with new product stack, but frankly I just don't think it would be viable here because pricing to do it properly is a multiple of LE-preferred ranges. You're honestly better off just buying a much less expensive VM from us and any of the other solid providers here to make your own HA/replication stack for significantly less money. The bonus is you also get geographical diversity in case of nukes.

@servers_guru said:
up!

@suyadi92 said:

@CoolGeek said:
It's not all back. I have some VPSs working and some not still.

From their discord:

The network is back online with limited capacity after the initial backup line didn't fully resolve the issue. We've switched to a different line card and we are working on restoring the full capacity. Thank you for your patience. More updates to follow.

The issue ended up being somewhat complex. We've been relying on a Juniper MX chassis with redundant RE's, line cards, PSUs, etc for service windows and redundancy in our edge routing. Don't quote me exactly on this as networking is not my specialty, but we had one of our MPC's fail (so the early quick/easy fixes didn't help: wasn't optics, carrier, fiber, power-related, etc). Unfortunately how this failed ended up causing multiple issues within the entire chassis and the difficult decision was made to just move to the newer edge routers immediately as the uncertainty around time/parts/confidence of repair on the MX was enough to bite the bullet and put in the new edge units ahead of schedule. Luckily they had been pre-configured a week prior as we were already prepping to quietly roll services over to a new stack seamlessly. Unluckily this happened at around 6AM PST after most of the team was up to 2-5AM due to big elections and we've been slowly moving things around behind the scenes to prep for the upgrade and additional rack space. Frankly: caught with our pants down, textbook Murphy's law.

There has been a plan that was already in motion to slowly upgrade the entire WA network to match the capacity/equipment/design that we recently switched PA over to and are very happy with. It will allow us the ability to add a ton of beneficial features for new deployments and also add them for existing customers at no charge if they want to incur a maintenance window.

I'm truly sorry to our customers for the downtime, hassle, and failures on my part and over this weekend we'll be implementing some large changes (starting internally) to alleviate this to the best of our ability from lessons learned the hard way this morning. Eligible services (meaning: not you with a yearly) are also getting a bump in SLA credits for this event.

I've only been a Crunchbits customer for 2 days, but this response instills confidence in my purchase.

CheepCluck

Go write a book about it you nerds

RandomGuy12

@crunchbits said:

@FAT32 said:
My brain: Don't say it, don't say it
Me: The end is...

Too soon, junior

@Allay said:
Reason I think they may be shutting down is that everything's been out of stock for a while, their sales hasn't replied to me in over a month either

Stuff is out of stock because there are big changes (and I hope: well received specials) coming and I don't want people buying something only to be (rightfully) upset 2-4 weeks later that X or Y is available and they just bought Z. I also don't want my team doing a bunch of extra work to service change requests on week old deployments. Toss in a surge of existing customer custom orders/growth and my personal preference to service and take care of the ones who helped us grow before onboarding new customers. I wish I could do both simultaneously, but I just can't reasonably meet those needs yet. Additionally, after getting sick pulling ~1hr per night sleep for too many days straight I realized we (myself and my entire team) have been pushing too hard for too long and need to have a plan to reasonably continue operations being mindful of mental and physical health. Luckily this is pretty easily achievable and we already have some internal task items to deal with it, but just to shed some light on why new sales/onboarding have been taking a backseat.

Plus, I don't like not being able to interact with friends and customers (discord, LET, etc) more often in an 'unofficial capacity'. It gets too cold and corporate, and I do value a lot of the input we receive from regular chats about what everyone is doing with their hardware. You wouldn't believe how often someone says they're doing XYZ and then it clicks for us that we have a certain type of hardware that isn't a standard product but we could deploy for their use-case and save them money every month. In my opinion, that kind of attention to detail is one of the biggest reasons to go with a small provider like you'll find on LET.

Also: I think we quoted you like 2 or 3 build requests already, unless I'm mixing you up with someone else (very possible, there are email threads that look to be the same person but might not be).

@suyadi92 said:

Crunchbits website are also down

Unfortunately, we thought we removed all dependencies on third party stuff not loading breaking the website a few months ago. Either a commit was mistakenly reverted or we missed something--but our blog going down caused the whole website (which is hosted offsite and redundant in an effort to keep website/billing/discord/ip phones all separately available avenues to reach us) to 500 out. Embarrassing, honestly. Completely my fault as well, as we have these little admin things to sniff out, sort out, test properly, and fix up but I have not been giving everyone enough time to really follow through on those tasks and do follow-ups.

@haste0675 said:
so much for HA

HA as far as services you buy from us? None are, none were ever sold to you as HA. That is something we've been investigating offering with new product stack, but frankly I just don't think it would be viable here because pricing to do it properly is a multiple of LE-preferred ranges. You're honestly better off just buying a much less expensive VM from us and any of the other solid providers here to make your own HA/replication stack for significantly less money. The bonus is you also get geographical diversity in case of nukes.

@servers_guru said:
up!

@suyadi92 said:

@CoolGeek said:
It's not all back. I have some VPSs working and some not still.

From their discord:

The network is back online with limited capacity after the initial backup line didn't fully resolve the issue. We've switched to a different line card and we are working on restoring the full capacity. Thank you for your patience. More updates to follow.

The issue ended up being somewhat complex. We've been relying on a Juniper MX chassis with redundant RE's, line cards, PSUs, etc for service windows and redundancy in our edge routing. Don't quote me exactly on this as networking is not my specialty, but we had one of our MPC's fail (so the early quick/easy fixes didn't help: wasn't optics, carrier, fiber, power-related, etc). Unfortunately how this failed ended up causing multiple issues within the entire chassis and the difficult decision was made to just move to the newer edge routers immediately as the uncertainty around time/parts/confidence of repair on the MX was enough to bite the bullet and put in the new edge units ahead of schedule. Luckily they had been pre-configured a week prior as we were already prepping to quietly roll services over to a new stack seamlessly. Unluckily this happened at around 6AM PST after most of the team was up to 2-5AM due to big elections and we've been slowly moving things around behind the scenes to prep for the upgrade and additional rack space. Frankly: caught with our pants down, textbook Murphy's law.

There has been a plan that was already in motion to slowly upgrade the entire WA network to match the capacity/equipment/design that we recently switched PA over to and are very happy with. It will allow us the ability to add a ton of beneficial features for new deployments and also add them for existing customers at no charge if they want to incur a maintenance window.

I'm truly sorry to our customers for the downtime, hassle, and failures on my part and over this weekend we'll be implementing some large changes (starting internally) to alleviate this to the best of our ability from lessons learned the hard way this morning. Eligible services (meaning: not you with a yearly) are also getting a bump in SLA credits for this event.

I must check for that, you had given me a few quotes earlier but I don';t think I recieved one when I contacted youagain, maybe I missed the email back, I'll email again

yoursunny

@crunchbits said:
Frankly: caught with our pants down, textbook Murphy's law.

Blame @ehab pants puller.

CheepCluck

@yoursunny can you sum up all those walls of texts?

yoursunny

@CheepCluck said:
@yoursunny can you sum up all those walls of texts?

Crunchbits will send a hoodie to everyone affected.

zed

@rcy026 said:

@crunchbits said:

[lots of relevant text snipped]

I'm truly sorry to our customers for the downtime, hassle, and failures on my part and over this weekend we'll be implementing some large changes (starting internally) to alleviate this to the best of our ability from lessons learned the hard way this morning. Eligible services (meaning: not you with a yearly) are also getting a bump in SLA credits for this event.

I just have to point out that this is the way a provider should handle outages. No bullshit, own the mistakes, explain what went wrong, apologize and make sure it doesn't happen again. Well done!

Definitely looking forward to some stock being available due to how they handle themselves here, refreshing.

danblaze

I have to admit:

Every business has its own problems and areas for improvement, but Eric's responsible attitude is one of the things that made me stay.

Also as a vendor in some areas, I have learned a lot from the way Eric treats his customers.

FAT32

@FAT32 said:
My brain: Don't say it, don't say it
Me: The end is...

Well after all good things really ended