BuyVM Miami Storage Slab - Total cluster data loss

Comments

• mw Member

  February 2025

  @Francisco said:

  @raindog308 said:

  @Francisco said: We don’t use seagate anymore since this isn’t the first time we’ve had drives go pop in the night. The new nodes have all been WD and have been excellent.

  This is the way.

  HGST has also been excellent. The first batch of slab nodes we built in 2016 was 200 drives and we’ve had maybe 8 or so go bad over the years?

  Either way, we’ve been doing a full top down look over everything and will likely kick off full raid background checks.

  Francisco

  backblaze Q4 2024 dropped

  https://www.backblaze.com/blog/backblaze-drive-stats-for-2024/

  

  As you can see, there are two 12TB drive models driving the high AFR for the HGST drives. The HUH721212ALN604 model began showing signs of an increased quarterly AFR in Q1 2023 and the HUH721212ALE604 model followed suit in Q3 2024. Without these drive models, the 2024 AFR for HGST drive would be 0.55%.
• Francisco Top Host, Host Rep, Veteran

  February 2025

  @mw said: backblaze Q4 2024 dropped
  https://www.backblaze.com/blog/backblaze-drive-stats-for-2024/

  Always super interesting I never used any of their 12TB's. The 8TB HGST's I have in use are still basically perfect. They're 7 years old now w/o a single SMART error or realloc.

  Francisco
• TimboJones Member

  February 2025 edited February 2025

  @Francisco said:

  @mw said: backblaze Q4 2024 dropped
  https://www.backblaze.com/blog/backblaze-drive-stats-for-2024/

  Always super interesting I never used any of their 12TB's. The 8TB HGST's I have in use are still basically perfect. They're 7 years old now w/o a single SMART error or realloc.

  Francisco

  Plot twist: no smart tests ran or no monitoring to give errors.
• Francisco Top Host, Host Rep, Veteran

  February 2025

  @TimboJones said: Plot twist: no smart tests ran or no monitoring to give errors.

  Seagates are hit/miss if the smart is even the right values. I have a pile of drives that report read bytes in the 'read errors' column. Brand spanking new ones that clear badblocks. I even re-ran them a 2nd time through badblocks to see how retarded I was.

  I'd assume it's just a firmware issue, but still, confusing.

  HGST SMART has been good. The few bad drives the RAID cards have kicked out I ran through bad blocks and can almost always confirm they were being flakey (terrible speeds or bad sectors).

  Francisco
• davide Member

  February 2025 edited February 2025

  @Francisco said:
  I have a pile of drives that report read bytes in the 'read errors' column.

  I have this Hitachi Deskstar that produces CRC errors periodically:

  ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
    1 Raw_Read_Error_Rate     0x000b   100   100   016    Pre-fail  Always       -       0
    2 Throughput_Performance  0x0005   133   133   054    Pre-fail  Offline      -       102
    3 Spin_Up_Time            0x0007   124   124   024    Pre-fail  Always       -       533 (Average 634)
    4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       616
    5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       5
    7 Seek_Error_Rate         0x000b   100   100   067    Pre-fail  Always       -       0
    8 Seek_Time_Performance   0x0005   121   121   020    Pre-fail  Offline      -       35
    9 Power_On_Hours          0x0012   090   090   000    Old_age   Always       -       74963
   10 Spin_Retry_Count        0x0013   100   100   060    Pre-fail  Always       -       0
   12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       605
  192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       618
  193 Load_Cycle_Count        0x0012   100   100   000    Old_age   Always       -       618
  194 Temperature_Celsius     0x0002   166   166   000    Old_age   Always       -       36 (Min/Max 20/46)
  196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       5
  197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
  198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
  199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       6803
  

  it is on a ZFS pool with block parity, so far no parity errors left the disk to reach the file system. It would seem almost safe to neglect even without FS parity.

  
• DataRecovery Member

  February 2025

  @davide said:

  @Francisco said:
  I have a pile of drives that report read bytes in the 'read errors' column.

  I have this Hitachi Deskstar that produces CRC errors periodically:

  CRC errors frequently mean a cable issue.

  Much more concerning is this:

  ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
    5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       5
  196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       5
  

  I would either stop using this drive, or would use it only for something unimportant.

  Having that said, Hitachi HDDs are great, could be considered the best ones on the market.
• davide Member

  February 2025 edited February 2025

  @DataRecovery said:

  @davide said:

  @Francisco said:
  I have a pile of drives that report read bytes in the 'read errors' column.

  I have this Hitachi Deskstar that produces CRC errors periodically:

  CRC errors frequently mean a cable issue.

  I think these aren't ATA errors but internal errors no?

  Edit: okay they are ATA errors.