38.69$/yr VPS Storage KVM/3 TB Storage/1 GB DDr4/1core/Ignore DMCA/1x IPv4/Crypto or PP/

mountyPython · April 7

@Jamph said:
Yabs for those interested, only thing really changed from the default was creating 4gb swap.

Do you need a 4 GB swap? It is presumably a swap file and because of the size on /dev/sdb with reduced performance. I have a 512 MB swap file in the root of sda1 where the OS is installed. I didn't think any more would be worth taking up space.

Anyway, I get similar yabs values to you

fio Disk Speed Tests (Mixed R/W 50/50) (Partition /dev/sdb):
---------------------------------
Block Size | 4k            (IOPS) | 64k           (IOPS)
  ------   | ---            ----  | ----           ----
Read       | 335.00 KB/s     (83) | 4.85 MB/s       (75)
Write      | 355.00 KB/s     (88) | 5.10 MB/s       (79)
Total      | 690.00 KB/s    (171) | 9.96 MB/s      (154)
           |                      |
Block Size | 512k          (IOPS) | 1m            (IOPS)
  ------   | ---            ----  | ----           ----
Read       | 11.17 MB/s      (21) | 6.98 MB/s        (6)
Write      | 12.25 MB/s      (23) | 7.33 MB/s        (7)
Total      | 23.43 MB/s      (44) | 14.31 MB/s      (13)

In comparison the other storage VPS on STOR3 (I think) has better performance but I don't know why that is. I thought they were on similar hardware.

fio Disk Speed Tests (Mixed R/W 50/50) (Partition /dev/sdb1):
---------------------------------
Block Size | 4k            (IOPS) | 64k           (IOPS)
  ------   | ---            ----  | ----           ---- 
Read       | 8.91 MB/s     (2.2k) | 101.95 MB/s   (1.5k)
Write      | 8.95 MB/s     (2.2k) | 102.49 MB/s   (1.6k)
Total      | 17.87 MB/s    (4.4k) | 204.45 MB/s   (3.1k)
           |                      |                     
Block Size | 512k          (IOPS) | 1m            (IOPS)
  ------   | ---            ----  | ----           ---- 
Read       | 218.25 MB/s    (426) | 323.74 MB/s    (316)
Write      | 229.85 MB/s    (448) | 345.30 MB/s    (337)
Total      | 448.10 MB/s    (874) | 669.05 MB/s    (653)

Jamph · April 7

@mountyPython said:

@Jamph said:
Yabs for those interested, only thing really changed from the default was creating 4gb swap.

Do you need a 4 GB swap? It is presumably a swap file and because of the size on /dev/sdb with reduced performance. I have a 512 MB swap file in the root of sda1 where the OS is installed. I didn't think any more would be worth taking up space.

Anyway, I get similar yabs values to you

fio Disk Speed Tests (Mixed R/W 50/50) (Partition /dev/sdb):
---------------------------------
Block Size | 4k            (IOPS) | 64k           (IOPS)
  ------   | ---            ----  | ----           ----
Read       | 335.00 KB/s     (83) | 4.85 MB/s       (75)
Write      | 355.00 KB/s     (88) | 5.10 MB/s       (79)
Total      | 690.00 KB/s    (171) | 9.96 MB/s      (154)
           |                      |
Block Size | 512k          (IOPS) | 1m            (IOPS)
  ------   | ---            ----  | ----           ----
Read       | 11.17 MB/s      (21) | 6.98 MB/s        (6)
Write      | 12.25 MB/s      (23) | 7.33 MB/s        (7)
Total      | 23.43 MB/s      (44) | 14.31 MB/s      (13)

In comparison the other storage VPS on STOR3 (I think) has better performance but I don't know why that is. I thought they were on similar hardware.

fio Disk Speed Tests (Mixed R/W 50/50) (Partition /dev/sdb1):
---------------------------------
Block Size | 4k            (IOPS) | 64k           (IOPS)
  ------   | ---            ----  | ----           ---- 
Read       | 8.91 MB/s     (2.2k) | 101.95 MB/s   (1.5k)
Write      | 8.95 MB/s     (2.2k) | 102.49 MB/s   (1.6k)
Total      | 17.87 MB/s    (4.4k) | 204.45 MB/s   (3.1k)
           |                      |                     
Block Size | 512k          (IOPS) | 1m            (IOPS)
  ------   | ---            ----  | ----           ---- 
Read       | 218.25 MB/s    (426) | 323.74 MB/s    (316)
Write      | 229.85 MB/s    (448) | 345.30 MB/s    (337)
Total      | 448.10 MB/s    (874) | 669.05 MB/s    (653)

I have 4gb for running Kavita on it, fin dit makes it run better and allows my other apps to stay running.

Prob could do it with less.

Calin · April 12

Hello dear customers,

We received more reports in the evening of the 11th regarding issues with the /storage partition on our storage VPS. After investigation it appears that the partition has become faild but it's active and operating in read-only mode. This is a familiar error, but the circumstances this time are unusual. Despite the mdadm RAID failing, the partition on the node remains intact, even though no HDDs are defective.

Fortunately, due to the more intricate RAID configuration on STOR-4, we have metadata backups for all VPS on the node from the /storage partition. This means there's no risk of data loss. However, WE REQUEST URGENT/ASAP all customers to promptly back up any important data from /storage partition, as we will need to rebuild the partition. Only the /storage partition will sufered a complete reinstallation, the main partition (/)still be intact of the VPS.

Q: How much time do I have to MAKE BACKUP?

A: We're allow all customers 96 hours (4 days) time to make backup. Additionally, we'll grant a 2-week extension on the next billing date.

Q: What caused this error?

A: We're still investigating the exact cause of this incident. However, it appears that the configuration differences, particularly the buffer-cache setup on STOR-4, may have contributed. Here's the current configuration:

20x 18 TB HDD SAS RAID 60
1x 960 GB SSD SAS for boot
1x 960 GB SAS for mdadm buffer cache
2x 2 TB HDD SAS RAID 1 for mdadm metadata backup

For prevent and improvment VPS node we start make next changes:

We intend to remove the buffer-cache during the reconfiguration of /storage to prevent similar issues in the future.
We start fix I/O speed , we will use the old cache method used on STOR-1 / STOR-2 and STOR-3 which was very efficient, and since we had the nodes up and running in the last 6-7 months everything went perfectly without any data loss

PLEASE NO REBOOT/SHUTDOWN YOU VPS FOR NO PERMANENTLY DATA LOST

We start post more updates on our discord server > https://discord.gg/7cnRBWEW8u and on our website page > https://panel.ihostart.com/index.php?rp=/announcements

Thanks for understand and sorry for this!

Regards,
Calin - C.E.O iHostART

Calin · April 12

@mountyPython said: In comparison the other storage VPS on STOR3 (I think) has better performance but I don't know why that is. I thought they were on similar hardware.

>

Hello yes we found now problem , it's from mdadm cache-buffer.
Any user?Hosting provider what meet this wired problem?

It's a similar thing what we usage us , on our configuration https://superuser.com/questions/1701778/improve-mdadm-raid5-with-ssd-cache

Regards

kennis942 · April 12

I am a customer GREAT SUPPORT!
Invoice #3570
I could always use more ram, was about to pay for it!

Calin · April 12

@kennis942 said: I am a customer GREAT SUPPORT!

Hello ram upgrade it's just for 38$/year packages , not monthly packages , very sorry

Regards

Fperkiss · April 12

Any more of these for a broke student? ^^

Calin · April 12

@Fperkiss said: Any more of these for a broke student? ^^

>

Hello for now , but maybe coming soon we start added other limited stock , we waiting for delivery other 9x 16 TB HDDs

Regards

shruub · April 12

@Calin said:
Hello dear customers,

We received more reports in the evening of the 11th regarding issues with the /storage partition on our storage VPS. After investigation it appears that the partition has become faild but it's active and operating in read-only mode. This is a familiar error, but the circumstances this time are unusual. Despite the mdadm RAID failing, the partition on the node remains intact, even though no HDDs are defective.

Fortunately, due to the more intricate RAID configuration on STOR-4, we have metadata backups for all VPS on the node from the /storage partition. This means there's no risk of data loss. However, WE REQUEST URGENT/ASAP all customers to promptly back up any important data from /storage partition, as we will need to rebuild the partition. Only the /storage partition will sufered a complete reinstallation, the main partition (/)still be intact of the VPS.

Q: How much time do I have to MAKE BACKUP?

A: We're allow all customers 96 hours (4 days) time to make backup. Additionally, we'll grant a 2-week extension on the next billing date.

Q: What caused this error?

A: We're still investigating the exact cause of this incident. However, it appears that the configuration differences, particularly the buffer-cache setup on STOR-4, may have contributed. Here's the current configuration:

20x 18 TB HDD SAS RAID 60
1x 960 GB SSD SAS for boot
1x 960 GB SAS for mdadm buffer cache
2x 2 TB HDD SAS RAID 1 for mdadm metadata backup

For prevent and improvment VPS node we start make next changes:

We intend to remove the buffer-cache during the reconfiguration of /storage to prevent similar issues in the future.
We start fix I/O speed , we will use the old cache method used on STOR-1 / STOR-2 and STOR-3 which was very efficient, and since we had the nodes up and running in the last 6-7 months everything went perfectly without any data loss

PLEASE NO REBOOT/SHUTDOWN YOU VPS FOR NO PERMANENTLY DATA LOST

We start post more updates on our discord server > https://discord.gg/7cnRBWEW8u and on our website page > https://panel.ihostart.com/index.php?rp=/announcements

Thanks for understand and sorry for this!

Regards,
Calin - C.E.O iHostART

woa woa, way too professional!

Jamph · April 12

I noticed the read write issue prior to the message which was sent so looks like I'm sol as server won't boot lol. My own fault for trying to fix my server at 1ish on the morning.

Luckily the server only host 'back ups' of the physical comics I own (2TB, luckily most are backed up else where so want lose much)

Got to say impressed with the email sent and comes across well.

Jamph · April 17

Does anyone know if this issue is resolved yet?

As my panel shows 'User ID not found, please report to the Administrator to verify if your VPS has a valid User assigned to it or not'.

it's possibly a case of me jumping the gun.

shruub · April 17

@Jamph said:
Does anyone know if this issue is resolved yet?

As my panel shows 'User ID not found, please report to the Administrator to verify if your VPS has a valid User assigned to it or not'.

it's possibly a case of me jumping the gun.

Best to DM @Calin on Discord or Telegram or some such if you haven't done so already.

mountyPython · April 17

My VPS went offline on 2024-04-15 at 00:59:44 EEST which is not 96 hours since the announcement and since then there's been no update on what is going on or how long it might take to complete.

ofera71 · April 17

@mountyPython You are lucky !
my VPS is offline for a week already , lol

Jamph · April 17

To be far, they did say 4 days to backup your data so any fixes wouldn't begin until day 5 which is today.

Then allow between 24-72 (at least) hour to fix it. So you are looking at around Friday/Saturday for the fix. With 2 weeks added on to the end of the billing cycle you are coming out with more time than paid for, so it's not all bad.

host_c · April 17

@shruub said: woa woa, way too professional!

you start to miss the basement hosting?

wii747 · April 17

I just got a VPS created email with information how to mount and format the /storage partition

tenpera · April 17

@wii747 said: I just got a VPS created email with information how to mount and format the /storage partition

which is?

Calin · April 17

@wii747 said: I just got a VPS created email with information how to mount and format the /storage partition

>

Hello yes we rebuild all VMs , in next hours we finish to add for all VMs the /storage partition!Unfortunetly this thing need make manual so take little bit time!

Thanks!

Regards

wil_123 · April 18

Why rebuild all VMs? The announcement stated only the storage partition is to be rebuilt and there should not affect the boot partition? Anyway all rebuilding the vps meaning all my previous works on the VM are lost!

@Calin said:

@wii747 said: I just got a VPS created email with information how to mount and format the /storage partition

>

Hello yes we rebuild all VMs , in next hours we finish to add for all VMs the /storage partition!Unfortunetly this thing need make manual so take little bit time!

Thanks!

Regards

ontheair · April 18

All data lost on /dev/sda (boot partition).
All data lost on /dev/sdb (storage partition).
IP changed.

How could that happen? Black magic?

Please let us know how to cancel and get refund. Thanks.

shruub · April 18

@host_c said:

@shruub said: woa woa, way too professional!

you start to miss the basement hosting?

Yeah man, I did kind of grew on me

shruub · April 18

@ontheair said:
All data lost on /dev/sda (boot partition).
All data lost on /dev/sdb (storage partition).
IP changed.

How could that happen? Black magic?

IHostMagic

sonontse · April 18

All our data magically disappear. Act of god. /s

kisiek · April 18

I'm done with that .
IP change data lost on boot /sda
If someone wants that machine I can transfer it for small fee .

Calin · April 18

@wil_123 said: Why rebuild all VMs? The announcement stated only the storage partition is to be rebuilt and there should not affect the boot partition? Anyway all rebuilding the vps meaning all my previous works on the VM are lost!

>
Hello, I preferred to install everything from 0, here I also mean the main node, so that there are no problems with the old configuration

On this process lost / , the 5 GBs partition

@kisiek said: IP change data lost on boot /sda

Yes, it has changed, it's still from the same /24 subnet, only so much that another one was allocated, it was a little more complicated to add the exact same IPv4 for almost 100 VMs

Regards

johndeo983 · April 18

@host_c Why don't you agree to help him? The situation here is out of control

zGato · April 18

@zGato said:

@Calin said:
3.It is recommended that this server not be used for production, only for backup / seedbox / torrent and others

= you know shit ain't gonna be stable.

warned y'all

Jamph · April 18

Yes losing data/having to backup can be a pain, but feel saying it's out of control is abit OOT.
Data loss can happen regardless who the provider is, It's way backups should be a part of your server set up process.

wii747 · April 18

I’m using this as as a 3rd backup of data as this service is very unreliable

Howdy, Stranger!

Categories

In this Discussion

38.69$/yr VPS Storage KVM/3 TB Storage/1 GB DDr4/1core/Ignore DMCA/1x IPv4/Crypto or PP/

Comments

Howdy, Stranger!

Quick Links

Categories

In this Discussion

38.69$/yr VPS Storage KVM/3 TB Storage/1 GB DDr4/1core/Ignore DMCA/1x IPv4/Crypto or PP/

Comments