Question about NIC direct connection between servers

liujason

Hi guys, I'm facing some problems about network interfaces setup, here is the detail:

I'm now building a small cluster with three nodes (debian 9), let's call them ServerA ServerB ServerC.
Each server have three physical network interfaces, let's call them NIC1 NIC2 NIC3. In these interfaces, NIC1/NIC2 are 10G ports and NIC3 is 1G port.
NIC3 of each server is connected to a 1G speed switch.

In order to have a higher speed for internal connection, I initially connect A1-B1 and A2-C1, it works fine with at a 10Gb speed. But the connection between ServerB and ServerC is depend on ServerA. So I connected B2-C2 then the network became unstable.

I know there is a loop problem for that so I bonded the A1-B1 and A2-C1 with a active-backup mode, I thought it can solve that problem, but not.

So is there a better way to solve this? Something like stp software?

rm_

liujason said: Something like stp software?

Yes, set up both NICs on each server into a bridge, then make sure it runs brctl stp brX on at each startup (via an up line in /etc/network/interfaces). The downside is that all 3 servers will have to go through one of them (and a random one, unless you tinker with "path costs") to access the 1G network. On the plus side is that when that one goes down, the network will auto-reconfigure to use a different one for that, though that could incur like half a minute of downtime during the process.

Here's how my setup with this looked like, when I used it like this:

Crossed out are the links auto-disabled by STP, basically the purpose of that is to prevent loops.