OVH Proxmox + pfsense + vrack + block ripe

floD

Hi,

I try to use pfsense in a KVM on my proxmox 4/5.
I use pfsense to catch one IP of my block of 16 IP's but the gateway of OVH does not respond to my pfsense.
My config seems to be correct. The same network config with a ubuntu VM works.

Tried with nic intel e1000 & virtio

• pfsense 2.4.2-RELEASE-p1
• proxmox 4 & 5 (tried on 2 hosts)

Steps:

• create a kvm and configure 2 nics to use the bridge of host proxmox binded on interface vrack.

• configure em0 with a public IP of my block ripe

  • IP from block
  • Default GW from block

• configure em1 for local network

With tcpdump, I see my the packets going to GW but not the respond

Thx for your help

Related post: https://www.lowendtalk.com/discussion/108864/help-with-ovh-proxmox-opnsense-pfsense

michaels

This is exactly the setup I run...
Does your virtual MAC in the OVH control panel match that of your WAN "link" in pfSense? I don't know why this isn't working for you though. I don't have vRack or anything like that. Happy to share any config you think might help.
M

Fidde

@michaels said:
This is exactly the setup I run...
Does your virtual MAC in the OVH control panel match that of your WAN "link" in pfSense? I don't know why this isn't working for you though. I don't have vRack or anything like that. Happy to share any config you think might help.
M

With vrack virtual mac isn't used, you get a "real" subnet with gw

Can you post screenshots of the pfsense config, blur parts of the ip if you want, but if you blur all of it we might not spot any error

EDIT: also post proxmox conf if you want

pbgben

Inside the pfsense web CP change these settings;

Interfaces > (wan name) > uncheck the two last boxes for Blocking bogen and private,

System > Routing > (Edit your gateway) > Tick last box to allow non-local GW

michaels

@fidde, my setup has been working for about 6 months. I was just offering to share the config in order to help @floD

floD

Hi, thx for your help

Proxmox conf

auto lo
iface lo inet loopback

iface eth0 inet manual

iface eth1 inet manual

iface eth2 inet manual

iface eth3 inet manual

auto vmbr1
iface vmbr1 inet static
    address  10.10.0.7
    netmask  255.255.224.0
    bridge_ports dummy0
    bridge_stp off
    bridge_fd 0
#post-up /etc/pve/kvm-networking.sh

# public
auto vmbr0
iface vmbr0 inet static
    address  178. ... .36
    netmask  255.255.255.0
    gateway  178. ... .254
    broadcast  178. ... .255
    bridge_ports eth0
    bridge_stp off
    bridge_fd 0
    network 178. ... .0

iface vmbr0 inet6 static
    address  2001:41D0:8:2D24::
    netmask  64
    post-up /sbin/ip -f inet6 route add ...
    post-up /sbin/ip -f inet6 route add ...
    pre-down /sbin/ip -f inet6 route del ...
    pre-down /sbin/ip -f inet6 route del ...

# vrack
auto vmbr2
iface vmbr2 inet static
    address  172.16.0.7
    netmask  255.240.0.0
    bridge_ports eth1
    bridge_stp off
    bridge_fd 0
    post-up route add -net 10.10.0.0 netmask 255.255.224.0 gw 172.16.0.7
    pre-down route del -net 10.10.0.0 netmask 255.255.224.0 gw 172.16.0.7

pfsense config

Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            37. ... .238      UGS         em0
37. ... .224/28   link#1             U           em0
37. ... .235      link#1             UHS         lo0
37. ... .238      da:82:af:ce:d9:18  UHS         em0
localhost          link#3             UH          lo0
172.16.0.0/12      link#2             U           em1
pfSense2           link#2             UHS         lo0

Internet6:
Destination        Gateway            Flags     Netif Expire
localhost          link#3             UH          lo0
fe80::%em0/64      link#1             U           em0
fe80::d882:afff:fe link#1             UHS         lo0
fe80::%em1/64      link#2             U           em1
fe80::4006:63ff:fe link#2             UHS         lo0
fe80::%lo0/64      link#3             U           lo0
fe80::1%lo0        link#3             UHS         lo0

• WAN : 37. ... .235
• LAN : 172.16.1.21
• Routing : Use non-local gateway through interface specific route. is checked
• Interface > WAN : Block private networks and loopback addresses & Block bogon networks are unchecked

michaels

So this is my interface config on my proxmox server

The vmbr0 is my "public interface"

vmbr1 is the interface for my "internal" network

# The loopback network interface
auto lo
iface lo inet loopback

# for Routing
auto vmbr1
iface vmbr1 inet manual
        post-up /etc/pve/kvm-networking.sh
        bridge_ports dummy0
        bridge_stp off
        bridge_fd 0

# vmbr0: Bridging. Make sure to use only MAC adresses that were assigned to you.
auto vmbr0
iface vmbr0 inet static
        address 94.xx.xx.100
        netmask 255.255.255.0
        network 94.xx.xx.0
        broadcast 94.xx.xx.255
        gateway 94.xx.xx.254
        bridge_ports eth0
        bridge_stp off
        bridge_fd 0

Pfsense

Destination         Gateway             Flags           Use         Mtu         Netif           Expire
default             OVH-GW              UGS             1052122     1500        em0 
EXTIP(s)            link#1              UHS             0           16384       lo0 
EXTIP(s)            link#1              U               0           1500        em0 
OVH-GW/32           02:00:00:a3:9e:c3   US              6796577     1500        em0 
127.0.0.1           link#3              UH              1004        16384       lo0 
LANIP/24            link#2              U               42896023    1500        em1 
LANIP               link#2              UHS             0           16384       lo0

WAN: 94.xx.xx.100
LAN: 192.168.2.x

Interface > WAN : Block private networks and loopback addresses & Block bogon networks are checked

The only other thing I remember, and I am sure this is for routing traffic from the LAN-> WAN is:

To create a route up to 192.168.23.254 (your main OVH IP), on an interface having no IP in this range, I use the commands:

route add -net 192.168.23.254/32 -iface em0
route add default 192.168.23.254

The first line tell the firewall that IP address 192.168.23.254 is on the side of the em0 interface (em0 is my WAN interface), the second one use this address as the default gateway.

Install shellcmd into pfSense and add the two commands above, this will make it survive a reboot.

LAN Internet

Firewall -> NAT -> Outbound

Manual Outbound NAT rule generation. If it isn’t created automatically add a rule with the Interface of WAN, source of your internal IP (192.168.1.x/24) leave everything else as default and save.

Hope there is something in there of some use!

floD

WAN & LAN are bridged on vmbr2 and it's the interface of my proxmox in vrack.

The block ripe is routed in vrack

Proxmox:

• IP pub (vmbr0)
• IP LAN vrack (vmb2)

Pfsense:

• WAN : one ip of block ripe (proxmox vmbr2)
• LAN : one ip of vrack network (proxmox vmbr2)

I don't find a solution

Why the same network config works on a linux ubuntu and not on a ferebsd or pfsense ? That's a bug from the ovh router ? Maybe because the mac address are not real ?

michaels

What are you using the vRack for? Would it not be easier to have your IP's attached to the IP pub block? I don't use the vRack as I have no other servers with OVH, but I am guessing you do? OVH's routing is "interesting" to say the least.

floD

I catch different IP's of my bloc RIPE into my vrack with VM's.
I use this system to expose my services in HA.
I want to create two pfsense in mode HA.

frdt

Hi,
I also have the same issue on proxmox 5 cluster connected to an ovh vrack. Ubuntu VM can ping the gateway from my IP block RIPE but wan interface of pfsense can't .... Why ubuntu can and pfsens not with the same configuration ?
Do you have any solution ??
thx