Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


BuyVM storage block is down - Page 2
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

BuyVM storage block is down

2»

Comments

  • jlayjlay Member
    edited May 2021

    @jar said:

    @Tim_kwakman said: Could you share that perhaps? I was thinking about creating something similar to monitor mine, but if it already exists and you are willing to share it then that would be great.

    Seems my memory failed and I apparently rewrote it to look for evidence in the Dovecot log. Either way it's a good bit of the work done. It goes with this cron job alongside it:

    0 0 * * * /usr/bin/rm -f /root/read_only_alerted_today

    https://clbin.com/8HPkv

    FYI Jar you can observe the mount status in the 'mounts' file in /proc (not typing the proper path because CloudFlare gets weird)

    Nothing wrong with monitoring the things dependent on the filesystem, but assuming it's actively being used - the kernel will auto-RO after a set time/number of write failures

    Systemd mount units are nice for handling service/filesystem inter-dependency

    edit: Depending on the distribution in use, you may have 'free' Mount units from the fstab generator thing

  • Tr33nTr33n Member

    If the filesystem is in read-only mode, and assuming /var/log is on the same partition, would such a log entry even exist if the filesystem is already read-only? I almost think not.

    I would also go the /proc/mounts route.

  • cybmp3cybmp3 Member

    @yoursunny said:

    Hello,

    There's a power outage in Luxembourg, causing the corruption of some storage blocks.
    Sorry for the trouble.
    Your bandwidth has been doubled.

    Thanks, LowEndSupport

    Actually...Someone told me that Las Veags is down too.....

  • FranciscoFrancisco Top Host, Host Rep, Veteran

    @cybmp3 said: Actually...Someone told me that Las Veags is down too.....

    No issues in Vegas or New York. Both datacenters are doubled cabled. I personally handled LV and I've had no less than 3 techs confirm (with photos) that things are correct in NY.

    The only bad thing going on with Slabs in Vegas & NY is we're basically sold out :)

    Francisco

  • What does double cabling look like?

    Is this a scheme to have redundant UPS's for a device or what exactly?

  • skorousskorous Member

    @sidewinder said:
    What does double cabling look like?

    Is this a scheme to have redundant UPS's for a device or what exactly?

    Each machine has two power supplies each coming from a different power source. If you lose one power source or one power supply you the other remaining compensates.
    Actually doesn't have to be two but that's pretty common.

  • jsgjsg Member, Resident Benchmarker

    @Francisco said:
    ... When we deployed these nodes we told ROOT that they must be double cabled but they ignored it, it seems.

    Bloody incredible. How many months worth of colo fees will you get for compensation?

    Plus: Which DC?

  • FranciscoFrancisco Top Host, Host Rep, Veteran

    @jsg said: Bloody incredible. How many months worth of colo fees will you get for compensation?
    Plus: Which DC?

    We're with Root, which hosts inside luxconnect.

    We'll get nothing, as is what happens with most SLA worthy issues with datacenters.

    I think in all the facilities I've worked with I've been able to claim a single SLA and they fought me the entire way. We just eat it. We issue credits as requested and try to make sure whatever happened, doesn't happen again.

    On the bright side, GTT finally gave me an install date!

    Francisco

    Thanked by 2jsg MannDude
  • jsgjsg Member, Resident Benchmarker

    @Francisco said:
    We're with Root, which hosts inside luxconnect.

    OK, noted "Root is a bunch of careless imbeciles".

    We'll get nothing, as is what happens with most SLA worthy issues with datacenters.

    So DC SLAs basically are just a marketing gadget?

  • FranciscoFrancisco Top Host, Host Rep, Veteran

    @jsg said: So DC SLAs basically are just a marketing gadget?

    dedis' might be a bit better but in colo you'll usually just get the SLA based on whatever broke.

    The network went down and qualifies for SLA? OK, they'll pick out whatever your BW bill is every month and SLA on that, not on the whole invoice. It's fine, but you end up going through a lot of effort, ruffle a lot of feathers, to get some lunch money.

    Some hosts will hold it against you that you claimed SLA, seen that more than a few times.

    Francisco

  • FranciscoFrancisco Top Host, Host Rep, Veteran

    @jsg said: OK, noted "Root is a bunch of careless imbeciles".

    See, that's the thing, they aren't.

    To date I've sent 4 large shipments to them, 2 crates, and 2 rounds of large boxes full of pizza box nodes. They handled all of it with care, racked it all perfectly, with the only issues coming up being ASROCK's QA being terrible and me not catching buggy hardware before it got crated.

    They handled my crate shipments very well, wired perfectly for everything except those slab nodes.

    They're a good facility. I like their staff a lot. This is just a misstep that just so happened to swing violently into my ballsack.

    Francisco

  • jsgjsg Member, Resident Benchmarker
    edited May 2021

    @Francisco

    "SLAs" - it seems the DC crowd needs a solid reminder ...

    "careless imbeciles" - I get your point, but still ... If that was my operation you'd have a memo on your table telling you that the culprit got fired (plus of course a sincere apology).

    Plus, forgive me for being less generous than you, but mistake != mistake. There are mistakes than can be understood and forgiven but forgetting to have a double power feed wired is something that should make the alarm go off in everybody's head in the DC. If not properly wiring customer equipment is a forgivable mistake in a DC then that DC is rotten.

  • FranciscoFrancisco Top Host, Host Rep, Veteran
    edited May 2021

    @jsg said: Plus, forgive me for being less generous than you, but mistake != mistake. There are mistakes than can be understood and forgiven but forgetting to have a double power feed wired is something that should make the alarm go off in everybody's head in the DC. If not properly wiring customer equipment is a forgivable mistake in a DC then that DC is rotten.

    For sure, I get that. We had a tech destroy one of our Ryzen CPU's the other day while doing a motherboard swap in New York. I warned them that it was pins and that they had to remove the CPU from the heatsink once they unscrew it, just because it gets stuck to it.

    Basically, just be careful, flat head screw driver easily pops the CPU off the bottom of the heatsink without a bit of damage.

    Well, they removed it fine. They then tried to install the cpu back into the new motherboard....while it was on the heatsink. No idea how that works given the lever would be in the way, I figure they were hammering on it not noticing the lever is there.

    Francisco

  • yoursunnyyoursunny Member, IPv6 Advocate

    @jsg said:
    forgetting to have a double power feed wired

    Double power feed is overrated.
    It's wasting copper (power strips) for no reason.

    Applications should be designed to have redundancy at higher levels.
    If power failure knocks out a network storage, just read the files from another storage!

  • jsgjsg Member, Resident Benchmarker

    @Francisco

    I wonder how such careless imbeciles stay employed. Maybe management failed to tell them "If you harm us we can and will talk; if you harm customers or their equipment you'll be fired, period".

    @yoursunny said:

    @jsg said:
    forgetting to have a double power feed wired

    Double power feed is overrated.
    It's wasting copper (power strips) for no reason.

    Applications should be designed to have redundancy at higher levels.
    If power failure knocks out a network storage, just read the files from another storage!

    BS on multiple levels. Try the 'maniacally smart youngster explains the world to grown up engineers' again once you have ten more years of real world experience under your belt.

  • The moment a guy came on Discord and said "Does Andy still work there? The one who trips in cables" I knew this is a fun place to work at.

  • @yoursunny said:

    @jsg said:
    forgetting to have a double power feed wired

    Double power feed is overrated.
    It's wasting copper (power strips) for no reason.

    Applications should be designed to have redundancy at higher levels.
    If power failure knocks out a network storage, just read the files from another storage!

    No reason? That's serious cheapskate talk and you should never be allowed to work in IT for a company. That's single user attitude, that shit doesn't fly at companies with many people who are paid 6 figures to sit around looking at 404's or having to go to backups. Continuous operations is more important than a fucking power strip by leagues.

    You are penny wise and pound foolish. ($999 virmach)

    Thanked by 2jsg notarobo
  • TonyBTonyB Member

    @yoursunny said:

    @jsg said:
    forgetting to have a double power feed wired

    Double power feed is overrated.
    It's wasting copper (power strips) for no reason.

    Applications should be designed to have redundancy at higher levels.
    If power failure knocks out a network storage, just read the files from another storage!

    I suspect you're not aware of what A/B power typically encompasses. The idea is it's two completely different power systems. It's not just there to protect you from a PSU failure on servers but failures much further up. If we're going to assume no A/B power then we also cannot assume each individual rack would not be affected by the same power disruption. This would mean to protect against any power issues we'd be looking at a geo-replicated storage system. This would be significantly more expensive and complicated and for most use cases completely unnecessary.

    Thanked by 2bulbasaur notarobo
  • FranciscoFrancisco Top Host, Host Rep, Veteran

    @TonyB said: This would be significantly more expensive and complicated and for most use cases completely unnecessary.

    We'd also be billing by the iop due to latency :P

    Francisco

    Thanked by 2bulbasaur vimalware
  • @yoursunny said:

    @jsg said:
    forgetting to have a double power feed wired

    Double power feed is overrated.
    It's wasting copper (power strips) for no reason.

    Applications should be designed to have redundancy at higher levels.
    If power failure knocks out a network storage, just read the files from another storage!

    ZFS is designed to be quite robust yet I managed to lose my data, so having redundancy at the application level isn't necessarily enough (granted, it was only a single vdev but still).

    Besides, more redundancy is always a good thing.

  • MaouniqueMaounique Host Rep, Veteran

    @TimboJones said: many people who are paid 6 figures to sit around looking at 404's

    I smiled. Very good picture.

Sign In or Register to comment.