New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Us1.workspace issue
Intelpentiumm
Member
in Outages
The us1.workspace is utter slow today. Can't access webmail.
Is there anyone facing email issues with us1 today?
Comments
We updated SmarterMail and at midnight it seems to have kicked off reindexing every mailbox on the server. I've been doing what I can to slow it down/spread out the reindexing process a bit to not impact the server.
I experienced this too, lots of delayed notification emails
In the last couple hours, SmarterMail behavior is the worst I've seen yet.
Pushed forward the software release again, trying to clear the delivery queue to mailboxes.
Seems to be behaving a bit better at the moment
Can you inform us in advance before pushing any software updates? This is getting annoying.
I would recommend doing updates on Friday evening or Saturday morning, so that only sys-admin's days get ruined, not the business clients.
Updates were run Friday/Saturday evening, the issue was the 12am cron that the software kicked off on Monday morning.
Would appreciate a heads up next time. Thanks.
Things break down, it happens.
If that can't be prevented in the future - it is how it is.
Still, it would be nice to make a status page.
I had clients call today and was as confused as they.
My contact email is tied to my pay pal so can't be changed and I can't be reached over email in case the service goes down because it's hosted there too (a bit of irony
).
Luckily nobody not a customer has any idea wtf us1.workspace is!
For me, an acceptable compromise would be to agree on times of day or week when it's best not to use the system. If I had known that early Monday mornings can be bad, I would have avoided it by going away and doing something else. Sounds crazy perhaps, but I since I have the luxury of planning around expected downtime, I would like to do so. I just need to know what and when to plan.
Sg-shared01 (singapore shared from namecrane) too seems to be down. Is it working there?
Edit - it seems to be back up again
It's down for me too.
The DA hosting SG location is very unstable and down very often.
Updates on Friday is the absolute stupidest idea, ever.
Sure. I agree. Saturday evening is probably better for the business stuff, globally.
When nobody is working? There's a reason Microsoft calls it "patch Tuesday".
Yes, Microsoft is a shiny example of caring for the users first.
Yes, because they have the highest usage in consumers and businesses. They've also made all the mistakes to learn from.
At work we refer to Friday's as "read-only Friday." Same goes for the entire month of December. There's always someone who learns those phrases the hard way and loses an entire weekend.
I propose read-only 2026. I already CBA for next year.
SmarterMail went down tonight.
Yes. Not working for me too
All sorted at this point.
We had ordered 2 new m2 SATA OS drives when we did the big RAM upgrade a month or so ago. Be it we lost the silicon lottery or whatever, but both failed (one failed a week or so ago, and then tonight the other).
My hope was taking all the stress off the drives would keep things going but that wasn't the case. Truthfully the only 'stress' these drives had was NGINX logs, so i'm not even sure why they nuked.
Fiberhub went in and got the node back online and we were able to reinstall and restore all service. Very happy with them doing that for us given its the holidays.
We'll need a proper maintenance window sometime in January to get things fully sorted, but for now things are humming.
While sorting that we may have figured out why we were having nightly burps for a couple minutes. SM hasn't been able to pinpoint it themselves, but we'll see in the next day or two.
Please let me know if you see otherwise.
Francisco
Some option to add high availability , add another machine to have a cluster ? or some figure that if some machine fail, not have everything down ?
>
We played with some sort of HA setup when we originally launched but it caused a lot of performance issues due to IO.
The big fault earlier in the month was software related so HA wouldn't have fixed that. They put out a bad update, so we downgraded as we've done in the past. Downgrading caused bigger issues though.
Ultimately they added a note to that release that downgrading is a BAD IDEA and will break things. If that notice was there at the start we wouldn't have attempted it.
Past that hardware hasn't been an issue at all. If it wasn't for a pair of crappy OS drives this wouldn't have been an issue either.
Still, we'll research to see what we can do w/o nuking our IO performance.
Francisco
Is it possible to setup a round-robin configuration to handle outgoing mail? If the main outgoing server is down, resend on backup server?
I guess such a feature would need to be optional, opt-in, maybe with a custom outgoing host. But for things like no-reply emails (the bulk of what I send), it would be perfect.
Namecrane also needs more MX records. Incoming mail should always have a fallback.
Explain the outgoing. This is on shared or cranemail?
We have 2 geo diverse mail servers for incoming and can add more when needed.
Francisco
Enabling SpamExperts gives you 4 MX records to add to your DNS.
I don't know about the rest of the infrastructure or running without that spam filter, but that's what I saw as a user.
I agree that sending out should be more stable for transactional emails - but not sure what the best way to handle that would be.
mx1.mxfilter.net and mx3.mxfilter.net both resolve to 23.147.8.11, while mx2.mxfilter.net and mx4.mxfilter.net both resolve to 162.244.95.10.
Although four MX records are published, they effectively map to only two unique mail hosts. From a redundancy and routing perspective, enabling SpamExperts results in two active endpoints, not four distinct servers.
This was done so we can slot in other servers when needed without requiring end user changes
Still, they’re in different regions.
Francisco