virmach - service in NYC migrating to new service provider with new IP addresses

bustersg

according to hetrixtools, it been down for 1 day 30 min.
however, the announcement ticket says "may result in 5-10 minutes downtime".

anybody in the same boat as me? (i will wait patiently if is not only me)
tried to reply to their migrating ticket but so far no response.

MallocVoidstar

NYC - Servers have been powered down for scheduled maintenance/migration.
Update 3PM PST: The servers and networking equipment have mostly been moved, and ports are being connected.
Update 3:17PM PST: All network switches are powered up and networking is being configured. Servers are being wired.
Update: 7:34PM PST: Wiring has been completed, and services are powered back on, but there will unfortunately be a delay in networking going online.

bustersg

@MallocVoidstar said:

NYC - Servers have been powered down for scheduled maintenance/migration.
Update 3PM PST: The servers and networking equipment have mostly been moved, and ports are being connected.
Update 3:17PM PST: All network switches are powered up and networking is being configured. Servers are being wired.
Update: 7:34PM PST: Wiring has been completed, and services are powered back on, but there will unfortunately be a delay in networking going online.

Thanks
Can sleep without worrying

commercial

Same boat, same issue
Being virmatched is a privilege
Can't wait for the next surprise ;p

TimesAndPlaces

VirMach NYCB is down quite frequently, ngl.

Last time NYCB was literally on fire with downtime of over 3 days. and that was... a couple months ago?

fluffernutter

Dedipath went under, so Virmach had to switch upstreams, just like a lot of LE* hosts here.

pepa65

@bustersg said:
anybody in the same boat as me? (i will wait patiently if is not only me)
tried to reply to their migrating ticket but so far no response.

I've been in that boat in LA for 3 weeks now, and now Chicago is down (as yet unreported on their Network Status, even though I did report the node).

And of course no response, ticket system timing out etc. as is usual for many months. They will buckle this year I feel.

bustersg

i still waiting my NYC node to be up.
hosting a wordpress in there, other services not critical.
am think of moving the wordpress out coz don't think is healthy for search engines to find the site down so long and i losing the ads revenue.

alilet

@bustersg said:
i still waiting my NYC node to be up.
hosting a wordpress in there, other services not critical.
am think of moving the wordpress out coz don't think is healthy for search engines to find the site down so long and i losing the ads revenue.

I think you missed a step where we take backup and migrate site to another server.

bustersg

@alilet said:

@bustersg said:
i still waiting my NYC node to be up.
hosting a wordpress in there, other services not critical.
am think of moving the wordpress out coz don't think is healthy for search engines to find the site down so long and i losing the ads revenue.

I think you missed a step where we take backup and migrate site to another server.

i can use my daily backup. just that my wordpress is docker..
don't want to mix container deployment with flat Nginx in 1 server
that is why i still hanging onto virmach as my container deployment

user_2million

I'm not in the same boat as you, but I'm in the Seattle boat and it has also sunk.

bustersg

Update 9/5 10:25AM PST: All servers except NYCB019, NYCB044, NYCB043, and NYCB004S are ready and we will begin assigning new IP addresses soon.

commercial

@bustersg said:
Update 9/5 10:25AM PST: All servers except NYCB019, NYCB044, NYCB043, and NYCB004S are ready and we will begin assigning new IP addresses soon.

3 days down ...
I'm on NYCB044, lucky me ;p
I can access to the VPS with VNC, but without outside network. Weird

yoursunny

bustersg

@commercial said:

@bustersg said:
Update 9/5 10:25AM PST: All servers except NYCB019, NYCB044, NYCB043, and NYCB004S are ready and we will begin assigning new IP addresses soon.

3 days down ...
I'm on NYCB044, lucky me ;p
I can access to the VPS with VNC, but without outside network. Weird

i just vnc to the server via virmach panel.
ipconfig show old ip still.
i tried 'reconfigure network' - no use
guess, they are still apply the new ips (phrase)

zGato

@bustersg said:

@commercial said:

@bustersg said:
Update 9/5 10:25AM PST: All servers except NYCB019, NYCB044, NYCB043, and NYCB004S are ready and we will begin assigning new IP addresses soon.

3 days down ...
I'm on NYCB044, lucky me ;p
I can access to the VPS with VNC, but without outside network. Weird

i just vnc to the server via virmach panel.
ipconfig show old ip still.
i tried 'reconfigure network' - no use
guess, they are still apply the new ips (phrase)

They need to get their BGP thingy signed and stuff still. ISPs that don't use secure BGP routes should be able to get a ping to their network, but the ones that do need to wait.

dev_vps

https://billing.virmach.com/index.php?rp=/store/account-support-level

Priority Support is $300 per year
Premium Support is $ 1200 per year
Critical Enterprise Support is $9,600 per year

chihcherng

@bustersg said:

@commercial said:

@bustersg said:
Update 9/5 10:25AM PST: All servers except NYCB019, NYCB044, NYCB043, and NYCB004S are ready and we will begin assigning new IP addresses soon.

3 days down ...
I'm on NYCB044, lucky me ;p
I can access to the VPS with VNC, but without outside network. Weird

i just vnc to the server via virmach panel.
ipconfig show old ip still.
i tried 'reconfigure network' - no use
guess, they are still apply the new ips (phrase)

This is what I do to get one of my Virmach VPSes online. VNC into that server. Edit /etc/network/interfaces to use my new IP. Reboot and test. You need to guess the default gateway IP. I use the IP ending with 1.

I have 2 Virmach VPSes. I don't need to do anything for the other one.

bustersg

@chihcherng said:

@bustersg said:

@commercial said:

@bustersg said:
Update 9/5 10:25AM PST: All servers except NYCB019, NYCB044, NYCB043, and NYCB004S are ready and we will begin assigning new IP addresses soon.

3 days down ...
I'm on NYCB044, lucky me ;p
I can access to the VPS with VNC, but without outside network. Weird

i just vnc to the server via virmach panel.
ipconfig show old ip still.
i tried 'reconfigure network' - no use
guess, they are still apply the new ips (phrase)

This is what I do to get one of my Virmach VPSes online. VNC into that server. Edit /etc/network/interfaces to use my new IP. Reboot and test. You need to guess the default gateway IP. I use the IP ending with 1.

I have 2 Virmach VPSes. I don't need to do anything for the other one.

does not work for me on the gw part.
i login and check.. etc/hosts already on new ip (though i clicked on reconfigure network)
but when i del existing and add new gw, i get error says gw ip cannot be reach.
tried xx.xx.xx.1 and xx.xx.1.1 both also cannot add.

i can't ping 8.8.8.8 on my vps.
also ipconfig ens3 still show old ip

bustersg

Guys VPS up online

Follow this guide to change IP - https://linuxhint.com/change-ip-address-ubuntu/
the recofigure network does not work ( not for me) maybe all it does is update etc/hosts
but we also need update the network file

im on ubuntu so $ sudo vi /etc/netplan/*.yaml
1) update IP
2) gateway to xx.xx.xx.1

Levi

Does virmach sent any notificiations to the affected clients or just plain old "they love to get virmached" attitude and 0 communication?

bustersg

@LTniger said:
Does virmach sent any notificiations to the affected clients or just plain old "they love to get virmached" attitude and 0 communication?

popup notice in control panel. wasnt there yesterday but has today.

by the way,
my laptop can ping / ssh new ip
server inside can ping 8.8.8.8
hetrixtool fail to ping new ip
my vps in provider A can ping new ip
my vps in provider B cannot ping new ip

so, new ip also need progration?

alilet

Probably because of invalid RPKI which needs to be fixed by their IP provider

https://rpki-validator.ripe.net/ui/46.233.44.0/23?validate-bgp=true

Void

@dev_vps said:
https://billing.virmach.com/index.php?rp=/store/account-support-level

Priority Support is $300 per year
Premium Support is $ 1200 per year
Critical Enterprise Support is $9,600 per year

Of course. How else is he supposed to make money? Selling $7/yr VPs ?

MallocVoidstar

@LTniger said:
Does virmach sent any notificiations to the affected clients or just plain old "they love to get virmached" attitude and 0 communication?

They sent at least one email, I got it on the 1st. I think they've sent at least 1 more regarding NY specifically but my servers are elsewhere.

jtk

@LTniger said:
Does virmach sent any notificiations to the affected clients or just plain old "they love to get virmached" attitude and 0 communication?

You sometimes get emails about things like upcoming IP address changes. Recently this happened due to their reliance on Dedipath, but if you only rely only on direct email and portal notifications you're often left in the dark about current status and what to do. For example, you might have gotten an email that states:

[...] we will be changing your server's main IP address and additional IP addresses. This maintenance is expected to occur today, Sunday, September 3rd [...] Your new IP address will not appear on your service details page until after maintenance is complete.

However, many (most?) VMs for which this email went out are currently unusable due to delays because new address assignment prefixes have not being fully delegated and properly announced. You might know and be OK with this if you carefully follow threads or random discussions here or in LES. And if you're paying < $10/year for a VM maybe this is the trade-off.

Virmach communicates extensively on LES. Many there seem to appreciate that and seem OK with the state-of-affairs. It is interesting if nothing else. Personally I'm bewildered how the now 1+ year of inconsistency and instability continues. The cashflow must still be good. Good for Virmach, but my contracts (less 100 but more than 10) all end soon and looking forward to just following the show from the sidelines.

dev_vps

@jmaxwell said:

@dev_vps said:
https://billing.virmach.com/index.php?rp=/store/account-support-level

Priority Support is $300 per year
Premium Support is $ 1200 per year
Critical Enterprise Support is $9,600 per year

Of course. How else is he supposed to make money? Selling $7/yr VPs ?

you are right .....

How much support fees BuyVM charges for $20/year VPS ?
Racknerd support is pretty fast and excellent even for $10/year VPS
Inception Hosting support has been excellent for Euro 15 / year VPS

commercial

All's fine for me with the new IP ...
Sounds like I moved to NYCB08
Of course I had to manage my dns and Hetrix monitoring setup, nothing complicated.

** A new Yabs to check the performance ...**

## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ##

Yet-Another-Bench-Script

v2023-09-06

https://github.com/masonr/yet-another-bench-script

## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ##

Thu Sep 7 03:03:56 PM UTC 2023

Basic System Information:

Uptime : 0 days, 0 hours, 8 minutes
Processor : AMD Ryzen 9 5950X 16-Core Processor
CPU cores : 3 @ 3393.622 MHz
AES-NI : ✔ Enabled
VM-x/AMD-V : ✔ Enabled
RAM : 3.8 GiB
Swap : 0.0 KiB
Disk : 98.1 GiB
Distro : Ubuntu 22.04.3 LTS
Kernel : 5.15.0-83-generic
VM Type : KVM
IPv4/IPv6 : ✔ Online / ❌ Offline

IPv4 Network Information:

ISP : OnyxAponte LLC
ASN : AS35913 DediPath
Host : Virtual Machine Solutions LLC
Location : New York, New York (NY)
Country : United States

fio Disk Speed Tests (Mixed R/W 50/50):

Block Size	4k (IOPS)	64k (IOPS)
Read	240.15 MB/s (60.0k)	310.93 MB/s (4.8k)
Write	240.78 MB/s (60.1k)	312.56 MB/s (4.8k)
Total	480.94 MB/s (120.2k)	623.50 MB/s (9.7k)
Block Size	512k (IOPS)	1m (IOPS)
------	--- ----	---- ----
Read	101.03 MB/s (197)	444.10 MB/s (433)
Write	106.40 MB/s (207)	473.68 MB/s (462)
Total	207.44 MB/s (404)	917.79 MB/s (895)

iperf3 Network Speed Tests (IPv4):

Provider	Location (Link)	Send Speed	Recv Speed	Ping
Clouvider	London, UK (10G)	59.5 Mbits/sec	85.7 Mbits/sec	67.1 ms
Scaleway	Paris, FR (10G)	827 Mbits/sec	368 Mbits/sec	73.3 ms
NovoServe	North Holland, NL (40G)	837 Mbits/sec	807 Mbits/sec	73.2 ms
Uztelecom	Tashkent, UZ (10G)	614 Mbits/sec	247 Mbits/sec	169 ms
Clouvider	NYC, NY, US (10G)	651 Mbits/sec	750 Mbits/sec	3.18 ms
Clouvider	Dallas, TX, US (10G)	67.3 Mbits/sec	176 Mbits/sec	59.0 ms
Clouvider	Los Angeles, CA, US (10G)	72.5 Mbits/sec	146 Mbits/sec	54.4 ms

Geekbench 6 Benchmark Test:

Test | Value
|
Single Core | 1732
Multi Core | 4062

Full Test | https://browser.geekbench.com/v6/cpu/2525904

Slow Disk, slow network ...
** The old Yabs was fare better**

fio Disk Speed Tests (Mixed R/W 50/50):

Block Size	4k (IOPS)	64k (IOPS)
Read	249.59 MB/s (62.3k)	102.34 MB/s (1.5k)
Write	250.25 MB/s (62.5k)	102.87 MB/s (1.6k)
Total	499.85 MB/s (124.9k)	205.22 MB/s (3.2k)
Block Size	512k (IOPS)	1m (IOPS)
------	--- ----	---- ----
Read	557.40 MB/s (1.0k)	637.82 MB/s (622)
Write	587.01 MB/s (1.1k)	680.30 MB/s (664)
Total	1.14 GB/s (2.2k)	1.31 GB/s (1.2k)

iperf3 Network Speed Tests (IPv4):

Provider	Location (Link)	Send Speed	Recv Speed	Ping
Clouvider	London, UK (10G)	498 Mbits/sec	466 Mbits/sec	71.4 ms
Scaleway	Paris, FR (10G)	busy	busy	79.8 ms
NovoServe	North Holland, NL (40G)	busy	837 Mbits/sec	82.9 ms
Uztelecom	Tashkent, UZ (10G)	791 Mbits/sec	353 Mbits/sec	160 ms
Clouvider	NYC, NY, US (10G)	917 Mbits/sec	929 Mbits/sec	2.50 ms
Clouvider	Dallas, TX, US (10G)	885 Mbits/sec	835 Mbits/sec	46.3 ms
Clouvider	Los Angeles, CA, US (10G)	869 Mbits/sec	584 Mbits/sec	59.6 ms

bustersg

the new ip is starting to get more accessibile now for me.
e.g. hetrixtools ping is up down up down.... vs couple hours ago was down flat line all the way,

bustersg

ok my website is up !!

bakageta

@jtk said:
You sometimes get emails about things like upcoming IP address changes. Recently this happened due to their reliance on Dedipath, but if you only rely only on direct email and portal notifications you're often left in the dark about current status and what to do. For example, you might have gotten an email that states:

[...] we will be changing your server's main IP address and additional IP addresses. This maintenance is expected to occur today, Sunday, September 3rd [...] Your new IP address will not appear on your service details page until after maintenance is complete.

However, many (most?) VMs for which this email went out are currently unusable due to delays because new address assignment prefixes have not being fully delegated and properly announced. You might know and be OK with this if you carefully follow threads or random discussions here or in LES. And if you're paying < $10/year for a VM maybe this is the trade-off.

Carefully follow random threads on a forum? The network status page is decently well updated, and should kinda be the first place you check when you've got an outage, especially one that you got an email about and is running longer than stated. I know everyone uses LET/LES as their personal support agent, but c'mon.

Ongoing:

NYC - Servers have been powered down for scheduled maintenance/migration.
Update 3PM PST: The servers and networking equipment have mostly been moved, and ports are being connected.
Update 3:17PM PST: All network switches are powered up and networking is being configured. Servers are being wired.
Update: 7:34PM PST: Wiring has been completed, and services are powered back on, but there will unfortunately be a delay in networking going online.
Update 9/5 10:25AM PST: All servers except NYCB019, NYCB044, NYCB043, and NYCB004S are ready and we will begin assigning new IP addresses soon.
Update 09/7 3AM PST: All new IP addresses have been assigned, except for NYCB004S which is still being worked on. New IP addresses will take some time to be fully functional globally.

Unexpected Outage:

There's a routing issue for the IPv4 subnets used by DENZ001, DENZ002, ATLZ005, ATLZ006.
Update: If routing issue is not resolved today, most services will be brought back online from backups.
ATLZ002, SEAZ004, SJCZ001, SJCZ002, SJCZ005, SJCZ006, SJCZ007, SJCZ001 and SJCZ008
Seattle - This location has unfortunately gone offline. We are still attempting to reach Sabey (the facility owner.)
Update: We are attempting to deploy new servers by the end of week and will load in from backups. 

I mean, you're aware of the outage, the cause, steps being taken, and a list of updates, what do you expect, hourly emails?