Configuring link based IPMP on Solaris

This post describes how to configure link based IPMP interfaces in Solaris. IPMP eliminates single network card failure and it ensures system will be always accessible via network.

In this example we will configure a link based IPMP using network interfaces ce0 & ce1.

  1. Confirm NICs and status:
    # dladm show-dev
    e1000g0         link: up        speed: 1000  Mbps       duplex: full
    ce0             link: unknown   speed: 1000  Mbps       duplex: full
    ce1             link: unknown   speed: 1000  Mbps       duplex: full
    # ifconfig -a
    lo0: flags=1000849 mtu 8232 index 1
            inet 127.0.0.1 netmask ff000000
    e1000g0: flags=1000843 mtu 1500 index 2
            inet 192.168.30.41 netmask ffffff00 broadcast 192.168.30.255
            ether 8:0:20:93:c9:af
    ce0: flags=1000842 mtu 1500 index 3
            inet 0.0.0.0 netmask 0
            ether 0:3:ba:56:b:51
    ce1: flags=1000842 mtu 1500 index 4
            inet 0.0.0.0 netmask 0
            ether 0:3:ba:56:b:52
  2. Add the required IP address (/etc/hosts) and netmask (/etc/netmasks)
    # grep 10.1.1.10 /etc/hosts
    10.1.1.10	ipmptest
    # grep 10.1.1 /etc/netmasks
    10.1.1.0	255.255.255.0
    
  3. Bring up (plumb) the NICs
    # ifconfig ce0 plumb
    # ifconfig ce1 plumb
    
  4. Add IP address and IP multipathing
    # ifconfig ce0 10.1.1.10 netmask 255.255.255.0 broadcast + group ipmp1 up
    # ifconfig ce1 group ipmp1 up
    
  5. Verify configuration
    # dladm show-dev
    e1000g0         link: up        speed: 1000  Mbps       duplex: full
    ce0             link: up        speed: 1000  Mbps       duplex: full
    ce1             link: up        speed: 1000  Mbps       duplex: full
    # ifconfig -a
    lo0: flags=1000849 mtu 8232 index 1
            inet 127.0.0.1 netmask ff000000
    e1000g0: flags=1000843 mtu 1500 index 2
            inet 192.168.30.41 netmask ffffff00 broadcast 192.168.30.255
            ether 8:0:20:93:c9:af
    ce0: flags=1000842 mtu 1500 index 3
            inet 10.1.1.10 netmask ffffff00 broadcast 10.1.1.255
            groupname ipmp1
            ether 0:3:ba:56:b:51
    ce1: flags=1000843 mtu 1500 index 4
            inet 0.0.0.0 netmask ff000000 broadcast 0.255.255.255
            groupname ipmp1
            ether 0:3:ba:56:b:52
  6. Verify IPMP failover with the following commands. Basically enable and disable interfaces using the if_mpadm command:
    # ip_mpadm -d ce0
    # ifconfig -a
    # ip_mpadm -r ce0
    # ifconfig -a
    # ip_mpadm -d ce1
    # ifconfig -a
    # ip_mpadm -r ce1
    # ifconfig -a
    
    All failovers are logged in /var/adm/messages. For example:
    # grep in.mpathd /var/adm/messages
    Apr 14 09:32:24 ipmptest in.mpathd[3800]: [ID 215189 daemon.error] The link has gone down on ce0
    Apr 14 09:32:24 ipmptest in.mpathd[3800]: [ID 594170 daemon.error] NIC failure detected on ce0 of group ipmp1
    Apr 14 09:32:24 ipmptest in.mpathd[3800]: [ID 832587 daemon.error] Successfully failed over from NIC ce0 to NIC ce1
    Apr 14 09:32:57 ipmptest in.mpathd[3800]: [ID 820239 daemon.error] The link has come up on ce1
    Apr 14 09:32:57 ipmptest in.mpathd[3800]: [ID 299542 daemon.error] NIC repair detected on ce0 of group ipmp1
    Apr 14 09:32:57 ipmptest in.mpathd[3800]: [ID 620804 daemon.error] Successfully failed back to NIC ce0
    Apr 14 09:46:59 ipmptest in.mpathd[3800]: [ID 832587 daemon.error] Successfully failed over from NIC ce1 to NIC ce0
    Apr 14 09:47:07 ipmptest in.mpathd[3800]: [ID 620804 daemon.error] Successfully failed back to NIC ce0
  7. Persistent configuration... To make our IPMP work across system reboots, we create configuration files for both the NICs. For example:
    # cat /etc/hostname.ce0
    10.1.1.10 netmask + broadcast + group ipmp1 up 
    # cat /etc/hostname.ce1
    group ipmp1 up