Understanding LLTSTAT output

The LLTSTAT command line tool can be used to display statistics of the Low Latency Transport (LLT) protocol that is used for the heartbeats of a VERITAS Cluster Server (tm) cluster. This information can be useful when troubleshooting potential interconnect issues on the heartbeat network between cluster nodes.

The LLT configuration file /etc/llttab describes the systems in the cluster and private network links among them. LLT runs in layer 2 of the network stack.

Typical output

Without applying any options lltstat outputs link statistics

root@solnod1# lltstat
LLT statistics:
    988803     Snd data packets         495884     Rcv data packets
    11400      Snd retransmit data      687        Rcv out of window
    28824991   Snd connect packets      0          Rcv duplicates
    1651       Snd independent ACKs     104        Rcv out of sequence
    494559     Snd piggyback ACKs       0          Rcv datagrams dropped
    0          Snd indenepdent NACKs    1481482    Snd loopback packets
    0          Snd Piggybank NACKs      
LLT errors:
    0          Rcv not connected        4          Snd not connected
    0          Rcv unconfigured         0          Snd no buffer
    0          Rcv bad dest address     0          Snd error drops
    0          Rcv bad source address   625        Snd no links up
    0          Rcv bad generation       0
    0          Rcv malformed packet     0

Note: There are three columns of data, followed by another three columns next to it

1. — contains the number of packets sent or received
2. — denotes either Snd (Data Sent) or Rcv (Data Received)
3. — provides a short description of each statistic (see below for detailed description)

List Peer Systems

Whilst, the lltstat command is used to interrogate the LLT protocol. The -n option displays the current status of the peer systems.

root@solnod1# lltstat -n
LLT node information:
    Node                 State    Links
   * 0 solnod1           OPEN        3
     1 solnod2           OPEN        3

root@solnod2# lltstat -n
LLT node information:
    Node                 State    Links
     0 solnod1           OPEN        3
   * 1 solnod2           OPEN        3

Note Each node has three links and that each node is in the open state. The asterisk (*) denotes the node on which you typed the command.

The -n option may be used with the -v or -vv (verbose and very-verbose options, respectively) to display link information and MAC addresses.

root@solnod1# lltstat -nv
LLT node information:
    Node                 State    Links
   * 0 solnod1           OPEN        3
     1 solnod2           OPEN        3
     2                   CONNWAIT    0
     3                   CONNWAIT    0
   :
   : {snip}
   :
    31                   CONNWAIT    0
root@solnod1# lltstat -nvv
LLT node information:
    Node                 State    Link  Status  Address
   * 0 solnod1           OPEN
                                  ce0   UP      08:00:20:B4:C5:68
                                  ce7   UP      08:00:20:B4:C4:01
                                  ce3   UP      08:00:20:B6:B5:D9
     1 solnod2           OPEN
                                  ce0   UP      08:00:20:B4:9E:36
                                  ce7   UP      08:00:20:B4:A2:6D
                                  ce3   DOWN
     2                   CONNWAIT
                                  ce0   DOWN
                                  ce7   DOWN
                                  ce3   DOWN
     3                   CONNWAIT
                                  ce0   DOWN
                                  ce7   DOWN
                                  ce3   DOWN
   :
   : {snip}
   :
    31                   CONNWAIT
                                  ce0   DOWN
                                  ce7   DOWN
                                  ce3   DOWN

Note: The output lists all possible 32 nodes of the cluster. It reports on the two nodes in the cluster, solnod1 and solnod2, plus non-existent nodes.

For each correctly configured node, the information should show a state of open, a status for each link of up, and an address for each link. However, the output in the example shows that for the solnod2 node the private network connection is possibly broken or the information in the /etc/llttab file may be incorrect.

LLT port status

To obtain information about the ports open for LLT, type lltstat -p on any node. For example:

solnod2# lltstat -p
LLT port information:
  Port   Usage      Cookie
  0      gab        0x0
         opens:     0 2 3 4 5 6 7 8 9 10 11 ... 60 61 62 63
         connects:  0 1 
  7      gab        0x7
         opens:     0 2 3 4 5 6 7 8 9 10 11 ... 60 61 62 63
         connects:  0 1 
  31     gab        0x1F
         opens:     0 2 3 4 5 6 7 8 9 10 11 ... 60 61 62 63
         connects:  0 1 

List Kernel Tunables

Use the lltstat -t on any node to list the LLT kernel tunables. For example:

solnod1# lltstat -t
LLT kernel tunables:
    maxnids: 32
    maxports: 32
    nominpad: 0
    rthreadpri: 98
    basetimer: 50000

Note: The basetimer is a timeout value of 50ms (50000 microseconds).

List Link information

The lltstat -l lists information about each configured LLT link. For example:

solnod1# lltstat -l
LLT link information:
   Link  Tag   State  Type  Pri     SAP    MTU    Addrlen
         Xmit          Recv          Err           LateHB
         Broadcast
   0     ce0   on     ether hipri   0xCAFE 1500   6
         4805041       96371727      0             109      
          FF:FF:FF:FF:FF:FF
   1     ce7   on     ether hipri   0xCAFE 1500   6
         4805041       96371727      0             114      
          FF:FF:FF:FF:FF:FF
   2     ce3  on      ether hipri   0xCAFE 1500   6
         4806098       76384154      0             122      
          FF:FF:FF:FF:FF:FF

Note: link 0 is ce0, link 1 is ce7 and link 2 is ce3. This should also correspond with what is in /etc/llttab.

List Configuration Directives

The lltstat -c to display LLT configuration directives. For example:

solnod1# lltstat -c
LLT configuration information:
    node: 1
    name: sunclu2
    cluster: 102
    version: 1.1
    nodes: 0 - 31
    max nodes: 32
    max ports: 32
    links: 3
    mtu: 1460
    max sdu: 66560
    broadcast HB: 1
    warning level: 20

If you simply want to display the cluster ID, you can pipe the output to grep. For example:

# lltstat -c | grep cluster
    cluster: 102

lltstat link statistics explained

The table below list expands on the lltstat link statistics