I recently rebooted my KVM server. When I do this, I often have to run a script to update Open vSwitch by removing unused ports, but this reboot messed up my entire iSCSI bond and vSwitch configuration. It looks like some of the control files for when you reboot the node were broken and needed to be repaired. Now, I am still not sure why these steps are required, but they do fix the problem for me.
jj
Normally, the following change to /etc/rc.d/rc.local is sufficient to fix my port issues, but not my bond issues:
/sbin/service openvswitch restart
portlist=`/bin/ovs-vsctl show|/bin/grep Port|/bin/grep vnet|/usr/bin/awk -F\" '{print $2}'`
for x in $portlist; do /bin/ovs-vsctl del-port ${x}; done
The above code simply restarts the Open vSwitch daemons and deletes any normal VNET ports so that when the virtual machines boot, the server is ready and clean. However, my real problem was due to my 10 Gb network bond’s being broken. That required a bit more to fix. Specifically, I needed to rebuild the bond’s configuration files, as the latest NetworkManager fix had removed critical elements related to Open vSwitch.
The fix was to modify the /etc/sysconfig/network-scripts/ifcfg-<iface> files for the interfaces (iface) that are part of the bond to add the following lines:
MASTER="bond0" SLAVE=yes NM_CONTROLLED=no
The fix was also to ensure that the following lines do not appear within those files:
DEVICETYPE=ovs TYPE=OVSPort OVS_BRIDGE=ovsbrX
Once you have done that (I did this for the ifcfg-eth2 and ifcfg-eth3 interfaces), you should verify that the ifcfg-bond0 file is correct.
DEVICE="bond0" ONBOOT=yes NETBOOT=yes BOOTPROTO=none NAME="bond0" DEVICETYPE=ovs TYPE=OVSPort OVS_BRIDGE=ovsbrX
In this way, the bond0 device will attach to the appropriate Open vSwitch device.
Now, all the above steps were for handling reboots. To handle changes on a running system, all I issued were the following commands:
ifdown bond0 ovs-vsctl add-bond ovsbrX bond0 eth2 eth3
Voilà, my Open vSwitch was fixed. Now, one thing we could do is look for error conditions within the ovs-vsctl show command used in /etc/rc.d/rc.local and, when we see them, fix them within that script as well. That would require using the previous few lines to rebuild the bond on the fly. This is accomplished using the following:
ovsscsi=0 /bin/ovs-vsctl show |/bin/grep bond0 if [ $? != 0 ] then ovsscsi=1 else /bin/ovs-vsctl show |/bin/grep bond0|/bin/grep error > /dev/null if [ $? = 0 ] then ovsscsi=1 fi fi if [ $ovsiscsi = 1 ] then /bin/ovs-vsctl del-br ovsscsi /bin/ovs-vsctl add-br ovsscsi /bin/ovs-vsctl add-bond ovsscsi bond0 eht2 eth3 fi
Unfortunately, it appears that for straight KVM Open vSwitch, even with RHEL 7 (CentOS 7), there is still a bit of automation required to get networking to survive a reboot.
One of these days, I plan on getting oVirt (RHEV) installed and working, but it does not install easily on just a single host.
