$100 if you fix this Proxmox VM issue

mw · April 6

$100 to the first person with a working fix

I have a server running Proxmox that refuses to give me networking on a VM. The vendor confirmed the VM's IP shows up in the ARP table but it cannot ping anything, not even the gateway

0-access-r02b08-drt-ams5.royalehosting.net(config)#show arp vrf internet <IP removed>

Address         Age (sec)  Hardware Addr   Interface
<IP removed>      0:00:51  bc24.116f.970e  Vlan100, Ethernet1/1

It's a brand new Proxmox install, all I did was create the VM and configure it the way all our other VMs are configured

Host:

root@host:~# cat /etc/network/interfaces
auto lo
iface lo inet loopback

auto vmbr0
iface vmbr0 inet static
        address 45.94.x.2/24
        gateway 45.94.x.1
        bridge-ports enp1s0f0np0
        bridge-stp off
        bridge-fd 0

iface enp1s0f0np0 inet manual

iface enp1s0f1np1 inet manual


source /etc/network/interfaces.d/*

VM:

ens18 is bridged to vmbr0

root@vm:~# cat /etc/network/interfaces
auto lo
iface lo inet loopback

auto ens18
iface ens18 inet static
        address 45.94.x.3/24
        gateway 45.94.x.1

It's a 9950X on a Supermicro board with 2x 40G NICs, nothing exotic.

I've tried:

Using the vendors provided prefix
Using our prefix
With and without LACP on the host
Reinstalling the host and VM
Swapping IPs between the host and VM
Bridging the 2nd 40G interface to vmbr1 and giving that to the VM
Praying
A CT container
virtio and Intel E1000 NICs
Changing MTU
Spoofing the host MAC's unused interface for the VMs NIC

Other boxes in this rack have VMs running fine. What could I be missing?

mw · April 6

If you want access to the box to give it a shot send me a PM, I'll give you credentials and whatever else you need to figure this out

barbaros · April 6

@emgh help this man, we can share the 100 bucks half and half.

wadhah · April 6

have you tried turning it on and off?

I'll take that 100$ in crypto thanks

georgedatacenter · April 6

We can help you...send us a private message.

Falzo · April 6

Can you use the additional IP on the host itself by just adding it to the interface.
Have you tried a routed setup?

mw · April 6

@Falzo said:
Can you use the additional IP on the host itself by just adding it to the interface.
Have you tried a routed setup?

On the host right now I assigned both its and the VMs IPs to the interface and they both work - the issue is getting the VM to work with it

I've not tried a routed setup, could you perhaps provide a config snippet so I can see what that would look like?

mw · April 6

@georgedatacenter said:
We can help you...send us a private message.

Sent - thanks!

nullnothere · April 6

@Falzo said: Can you use the additional IP on the host itself by just adding it to the interface.

This. At least verify that the IP is working.

I'd also leave a tcpdump running (only sniffing arps) to check what gives.

I vaguely remember you had some IPv6 issues as well - where everything looked right but somehow it didn't work. Seems similar. Not sure what the fix was there. Do you remember? Was the provider at fault (as in typo/fat-fingering, non-malicious stuff) for anything at all?

onidel · April 6

.

mw · April 6

@nullnothere said:

@Falzo said: Can you use the additional IP on the host itself by just adding it to the interface.

This. At least verify that the IP is working.

The IP works, using it on the host works, giving it to the VM doesnt - thats where the problem is

I'd also leave a tcpdump running (only sniffing arps) to check what gives.

Will give that a try

I vaguely remember you had some IPv6 issues as well - where everything looked right but somehow it didn't work. Seems similar. Not sure what the fix was there. Do you remember? Was the provider at fault (as in typo/fat-fingering, non-malicious stuff) for anything at all?

Could not get that fixed, eventually put it on ice. Perhaps I'll revisit it to see if it has anything to do with this issue

nullnothere · April 6

@mw said: On the host right now I assigned both its and the VMs IPs to the interface and they both work - the issue is getting the VM to work with it

Aah this is much better.

No iptables (or related) shenanigans going on - right? (never hurts to check).

mw · April 6

The VM can ping the host (since they share the same bridge device vmbr0) and the host can ping the VM - whatever is causing the problem is after the physical interface

Gonna try sniffing with tcpdump

mw · April 6

@nullnothere said:

@mw said: On the host right now I assigned both its and the VMs IPs to the interface and they both work - the issue is getting the VM to work with it

Aah this is much better.

No iptables (or related) shenanigans going on - right? (never hurts to check).

It's stock, but for completeness sake I did reset it. Made no difference

wdmg · April 6

Bind it to your VM, grab your VM MAC address on the primary interface, then ask your host what MAC they see in their arp table.

It's probably:

Hijacked/assigned elsewhere;
An old arp entry not cleared;
You misconfigured the VM or bridge (unlikely);
They're running a particular version of JunOS which has a hard ARP table size limitation and they need to downgrade/upgrade to fix it (unlikely);
They're running switch-level MAC filtering and that MAC isn't in the whitelist;

mw · April 6

@wdmg said:
Bind it to your VM, grab your VM MAC address on the primary interface, then ask your host what MAC they see in their arp table.

It's probably:

Hijacked/assigned elsewhere;

An old arp entry not cleared;

You misconfigured the VM or bridge (unlikely);

They're running a particular version of JunOS which has a hard ARP table size limitation and they need to downgrade/upgrade to fix it (unlikely);

They're running switch-level MAC filtering and that MAC isn't in the whitelist;

They confirmed they see the correct MAC in the ARP table, and I asked them about MAC filtering:

It’s not hard MAC filtering, it’s the arp table which can take some time to update
But I can manually clear it

Which they did and it did nothing - the vendor says its something server side, not their side

keoir · April 6

Happy to help out shoot me a dm with more info

nullnothere · April 6

@wdmg said:
Bind it to your VM, grab your VM MAC address on the primary interface, then ask your host what MAC they see in their arp table.

[...]

Yeah - I too suspect that somehow the VM's MAC isn't allowed/whitelisted and so it is dropping. IIRC, at other hosting providers (Hetzner, OVH), one gets an explicit MAC for the public IP which can then be assigned to the VM directly and bingo things work.

Not sure what the setup here is but it seems more provider guidance required.

@mw - you mention this exact setup works in your other machines at the same provider - so you didn't have to do any special MAC setup for those VMs with public IPs? If that's the case then as @wdmg points out - there's something odd about this particular port for filtering.

mw · April 6

@keoir said:
Happy to help out shoot me a dm with more info

Shot you a DM

mw · April 6

@nullnothere said:

@wdmg said:
Bind it to your VM, grab your VM MAC address on the primary interface, then ask your host what MAC they see in their arp table.

[...]

Yeah - I too suspect that somehow the VM's MAC isn't allowed/whitelisted and so it is dropping. IIRC, at other hosting providers (Hetzner, OVH), one gets an explicit MAC for the public IP which can then be assigned to the VM directly and bingo things work.

Not sure what the setup here is but it seems more provider guidance required.

@mw - you mention this exact setup works in your other machines at the same provider - so you didn't have to do any special MAC setup for those VMs with public IPs? If that's the case then as @wdmg points out - there's something odd about this particular port for filtering.

Yes we've never had to do anything special regarding MAC on the other machines running VMs in the same rack with the same prefix - the vendor says they triple checked and they do not have any filtering or security policy that would cause this issue

Alexgamma · April 6

myb the bridge vmbr0 isn’t properly connected or configured. Can you confirm if the tapXXX interface shows up in brctl show vmbr0 when the VM is running?

theraw · April 6

feel free to dm if still pending.

mw · April 6

Vendor tried disabling one of the interfaces on the switch to see if there was any weird looping, for some reason it took both of the hosts interfaces down

Right now we have one of the interfaces enabled on the switch, but both configured on the host to its two interfaces and both still work?

Very weird... anyway

@Alexgamma said:
myb the bridge vmbr0 isn’t properly connected or configured. Can you confirm if the tapXXX interface shows up in brctl show vmbr0 when the VM is running?

root@host:~# brctl show vmbr0
bridge name bridge id STP enabled interfaces
vmbr0 8000.248a07053416 no enp1s0f0np0

Does this show anything useful?

mw · April 6

@theraw said:
feel free to dm if still pending.

Sure, let me drop you a DM!

Falzo · April 6

@mw said:

@Falzo said:
Can you use the additional IP on the host itself by just adding it to the interface.
Have you tried a routed setup?

On the host right now I assigned both its and the VMs IPs to the interface and they both work - the issue is getting the VM to work with it

I've not tried a routed setup, could you perhaps provide a config snippet so I can see what that would look like?

add another interface with the main ip as address, no bridge ports or gateway and add the second IP address to it:

auto vmbr2
iface vmbr2 inet static
        address 45.94.x.2
        netmask 255.255.255.255
        bridge_ports none
        bridge_stp off
        bridge_fd 0
        bridge_maxwait 0

        up route add -host 45.94.x.3 dev vmbr2

put the VMs network on that bridge and use .3 as IP inside and .2 as gateway

I like setting things up always with host routes / pointopoint, but I think that's not needed here, as your IPs seem to be in the same subnet anyway.

Lince8 · April 6

I sended you a DM about.

unsafetypin · April 6

@Lince8 said:
I sended you a DM about.

Is this AI generated? This looks AI generated.

Edit: they removed the AI generated nonsense

Lince8 · April 6

@unsafetypin said:

@Lince8 said:
I sended you a DM about.

Is this AI generated? This looks AI generated.

Edit: they removed the AI generated nonsense

Yes, it's an answer given by the AI. I had a problem with proxmox too, and thanks to his solution I was able to solve the problem. It was just to try to help him, I don't need the reward he is offering

mw · April 6

@Falzo said:

@mw said:

@Falzo said:
Can you use the additional IP on the host itself by just adding it to the interface.
Have you tried a routed setup?

On the host right now I assigned both its and the VMs IPs to the interface and they both work - the issue is getting the VM to work with it

I've not tried a routed setup, could you perhaps provide a config snippet so I can see what that would look like?

add another interface with the main ip as address, no bridge ports or gateway and add the second IP address to it:

auto vmbr2
> iface vmbr2 inet static
>         address 45.94.x.2
>         netmask 255.255.255.255
>         bridge_ports none
>         bridge_stp off
>         bridge_fd 0
>         bridge_maxwait 0
> 
>         up route add -host 45.94.x.3 dev vmbr2
>

put the VMs network on that bridge and use .3 as IP inside and .2 as gateway

I like setting things up always with host routes / pointopoint, but I think that's not needed here, as your IPs seem to be in the same subnet anyway.

This works, the VM now has networking. It's not exactly what we want to do, but this is progress.

Does this working shed any light on why the way I usually do it doesn't work?

ilikebeans · April 6

@mw said:
$100 to the first person with a working fix

I have a server running Proxmox that refuses to give me networking on a VM. The vendor confirmed the VM's IP shows up in the ARP table but it cannot ping anything, not even the gateway
0-access-r02b08-drt-ams5.royalehosting.net(config)#show arp vrf internet <IP removed>

Address         Age (sec)  Hardware Addr   Interface
<IP removed>      0:00:51  bc24.116f.970e  Vlan100, Ethernet1/1
It's a brand new Proxmox install, all I did was create the VM and configure it the way all our other VMs are configured

Host:
root@host:~# cat /etc/network/interfaces
auto lo
iface lo inet loopback

auto vmbr0
iface vmbr0 inet static
        address 45.94.x.2/24
        gateway 45.94.x.1
        bridge-ports enp1s0f0np0
        bridge-stp off
        bridge-fd 0

iface enp1s0f0np0 inet manual

iface enp1s0f1np1 inet manual


source /etc/network/interfaces.d/*
VM:
ens18 is bridged to vmbr0
root@vm:~# cat /etc/network/interfaces
auto lo
iface lo inet loopback

auto ens18
iface ens18 inet static
        address 45.94.x.3/24
        gateway 45.94.x.1
It's a 9950X on a Supermicro board with 2x 40G NICs, nothing exotic.

I've tried:

Using the vendors provided prefix

Using our prefix

With and without LACP on the host

Reinstalling the host and VM

Swapping IPs between the host and VM

Bridging the 2nd 40G interface to vmbr1 and giving that to the VM

Praying

A CT container

virtio and Intel E1000 NICs

Changing MTU

Spoofing the host MAC's unused interface for the VMs NIC

Other boxes in this rack have VMs running fine. What could I be missing?

I HAD THIS EXACT ISSUE WITH THE EXACT SPEC YOUR HAVING, WITH THE EXACT SAME HOST!!!!

(although i had a 200G port & connectx6 mellanox NIC)

good luck fixing it, because i couldnt even after a reinstall. Royale wont be able to help...

GOOD LUCK!!!

is this by any chance the 45.94.221.0/24 range?

mw · April 6

@ilikebeans said:
@mw said:
$100 to the first person with a working fix

I have a server running Proxmox that refuses to give me networking on a VM. The vendor confirmed the VM's IP shows up in the ARP table but it cannot ping anything, not even the gateway
0-access-r02b08-drt-ams5.royalehosting.net(config)#show arp vrf internet <IP removed>

Address         Age (sec)  Hardware Addr   Interface
<IP removed>      0:00:51  bc24.116f.970e  Vlan100, Ethernet1/1
It's a brand new Proxmox install, all I did was create the VM and configure it the way all our other VMs are configured

Host:
root@host:~# cat /etc/network/interfaces
auto lo
iface lo inet loopback

auto vmbr0
iface vmbr0 inet static
        address 45.94.x.2/24
        gateway 45.94.x.1
        bridge-ports enp1s0f0np0
        bridge-stp off
        bridge-fd 0

iface enp1s0f0np0 inet manual

iface enp1s0f1np1 inet manual


source /etc/network/interfaces.d/*
VM:
ens18 is bridged to vmbr0
root@vm:~# cat /etc/network/interfaces
auto lo
iface lo inet loopback

auto ens18
iface ens18 inet static
        address 45.94.x.3/24
        gateway 45.94.x.1
It's a 9950X on a Supermicro board with 2x 40G NICs, nothing exotic.

I've tried:

Using the vendors provided prefix

Using our prefix

With and without LACP on the host

Reinstalling the host and VM

Swapping IPs between the host and VM

Bridging the 2nd 40G interface to vmbr1 and giving that to the VM

Praying

A CT container

virtio and Intel E1000 NICs

Changing MTU

Spoofing the host MAC's unused interface for the VMs NIC

Other boxes in this rack have VMs running fine. What could I be missing?
I HAD THIS EXACT ISSUE WITH THE EXACT SPEC YOUR HAVING, WITH THE EXACT SAME HOST!!!!

(although i had a 200G port & connectx6 mellanox NIC)

good luck fixing it, because i couldnt even after a reinstall. Royale wont be able to help...

GOOD LUCK!!!

is this by any chance the 45.94.221.0/24 range?

fr?

Howdy, Stranger!

Categories

In this Discussion

$100 if you fix this Proxmox VM issue

Comments

Howdy, Stranger!

Quick Links

Categories

In this Discussion

$100 if you fix this Proxmox VM issue

Comments