help needed for proxmox to proxmox gre tunnel

emre

recently I posted a discussion about the less amount of downtime when migrating customers from a proxmox node from a provider to another proxmox node in aother provider. (with moving the /24 ipv4 block between providers)

at the end of discussion with kind help from fellow let citizens the best way looks like creating a GRE TUNNEL between provider A and provider B and using ip addresess from provider A in provider B by the help of the GRE TUNNEL and after all the migrating done, Migrate the /24 from old AS# to new AS# in the new provider.

What I've done:

gre tunnel is easy. Is up and running in under 2 mins.
BUT to route ips currently used by KVM VM's in Provider A in Provider B gives a lot of headaches.

for not to mess up with the current production system I created a new debian Vm in provider A and set up the GRE tunnel between this VM and new Proxmox node on provider B (New proxmox node is empty)

created a vmbr2 bridge in Provider B proxmox node and assign the Gre Tunnel as bridge interfece to that.

After all the failed attempts ( Doing NAT with Real Ips etc, even Creating an openVPN tunnel instead of GRE, I am yet to be able to route any ip from Provider A VM to Provider B proxmox node)

I searched and searched, Trying to steer myself away of DUMB AI suggestions, Trying translating German, Russian, even Arabic ancient forum posts. But no Luck)

I failed miserably at the moment.

I need help about this.

Below is my current setup ( Ip addressess are imaginary) Can someone please point me to right direction?

Provider A:
Debian VM:
ip: 38.248.1.150
Gateway: 38.248.1.1

Provider B:
Empty proxmox Node:
ip: 185.17.1.2
Gateway: 185.17.1.1

Gre setup:
Debian VM:
ip tunnel add gre1 mode gre local 38.248.1.150 remote 185.17.1.2 ttl 255
ip link set gre1 up
ip addr add 172.18.185.57/30 dev gre1

Proxmox Node:
ip tunnel add gre1 mode gre local 185.17.1.2 remote 38.248.1.150 ttl 255
ip link set gre1 up
ip addr add 172.18.185.58/30 dev gre1

GRE TUNNEL UP AND RUNNING.

Proxmox Node:

auto vmbr2
iface vmbr2 inet static
address 38.248.1.151/24
gateway 38.248.1.1
bridge-ports gre1
bridge-stp off
bridge-fd 0

Promox Node: Migrated VM From Provider A:
Net0 Network Device: Bridge vmbr2

Migrated VM Ip: 38.248.1.152
Migrated VM Gateway: 38.248.1.1

---

Right here without any NAT nonsense I need to create a routing setup between my vmbr2 and and the GRE Interface.

This Is what I need help with.

Thank you all..

emre

PROBLEM SOLVED:
SOLUTION:

for those who interested after my original post with more trial and error here is the solution for my use case.

Provider A (Debian VM)            
Public IP: 38.248.1.150           
/24: 38.248.1.0/24 (in Provider A)
/24 Gateway: 38.248.1.1 (Provider A)

Provider B (Proxmox Node)
Public IP: 185.17.1.2
Gateway: 185.17.1.1

---

On Provider A (Debian VM)

# Install bridge utils if missing
apt install bridge-utils

# Create gretap tunnel
ip link add gre1 type gretap local 38.248.1.150 remote 185.17.1.2 ttl 255

# Bring tunnel up
ip link set gre1 up mtu 1450

# Create a bridge for the tunnel + upstream NIC
brctl addbr br0
brctl addif br0 ens18   # eth0 is your public interface in Provider A
brctl addif br0 gre1

# Assign Provider A’s public IP to the bridge (so Debian VM still works)
ip addr flush dev ens18
ip addr add 38.248.1.150/24 dev br0
ip link set br0 up
ip route add default via 38.248.1.1

---

On Provider B (Proxmox Node)
edit /etc/network/interfaces

auto gre1
iface gre1 inet manual
    pre-up ip link add gre1 type gretap local 185.17.1.2 remote 38.248.1.150 ttl 255
    up ip link set gre1 up mtu 1450
    post-down ip link del gre1

auto vmbr2
iface vmbr2 inet manual
    bridge_ports gre1
    bridge_stp off
    bridge_fd 0
    mtu 1476 # see below for extra notes

end edit and reboot

---

Migrate a VM to Provider A to Provider B
Migrated VM network settings - keep everything intact and original

---

IMPORTANT NOTE:

Because GRE uses more encapsulation overhead.
Lower the MTU in Provider B side
I set the MTU values as below and seems like working (may need a little bit more tweaking)

Provider B Proxmox node set vmbr2 MTU 1476 - already setup in /etc/network/interfaces
On Proxmox node side: Provider B Migrated VM: set net0 network device MTU 1400

---

EXTRA: Mandatory Tests:

From Migrated VM with ip address 38.248.1.152

 Host                             Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 38.248.1.1                     0.0%    11    1.3   1.2   1.0   1.8   0.2
 2. 62.113.192.83                  0.0%    11    1.5   1.6   1.2   2.0   0.2
 3. 62.113.192.91                  0.0%    11    1.1   1.4   1.1   1.8   0.2
 4. 80.150.168.241                 0.0%    11    2.3   2.2   1.8   2.4   0.2
 5. 62.157.248.2                   0.0%    11    1.3   1.5   1.3   1.8   0.2
 6. 212.156.101.219                0.0%    11   48.0  47.9  47.6  48.1   0.1
 7. 81.212.31.191                 20.0%    11   48.1  48.2  47.9  48.4   0.2
 8. 81.212.247.98                  0.0%    11   58.2  58.4  58.1  58.9   0.2
 9. 81.212.246.109                 0.0%    11   58.9  59.0  58.7  59.2   0.2
10. 195.175.103.254                0.0%    10   56.4  57.8  56.3  68.5   3.8
11. (waiting for reply)
12. 198.68.73.68                   0.0%    10   62.1  62.7  60.7  65.0   1.3

---

From My own office connection to Migrated VM

 Host                              Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 192.168.88.1                    0.0%    13    0.3   0.3   0.2   0.3   0.0
 2. (waiting for reply)
 3. 172.25.16.13                    0.0%    13    9.3   8.5   7.0  11.5   1.5
 4. 195.33.217.229                  0.0%    12   13.3  13.3  12.5  15.1   0.7
 5. 10.40.174.73                    0.0%    12   14.0  13.9  12.3  15.2   0.8
 6. (waiting for reply)
 7. (waiting for reply)
 8. (waiting for reply)
 9. 89.221.34.189                   0.0%    12   55.8  56.1  51.9  70.1   6.0
10. 62.67.110.46                    0.0%    12   79.2  63.0  60.0  79.2   5.2
11. 62.113.192.66                   0.0%    12   61.6  61.1  60.2  62.0   0.7
12. 38.248.1.152                    0.0%    12   61.4  62.2  60.8  66.7   1.6

host_c

Hmm

Nice, sincerely,

I would VXLAN it the next time, but if that works, +1