Understanding LLTSTAT output
The LLTSTAT command line tool can be used to display statistics of the Low Latency Transport (LLT) protocol that is used for the heartbeats of a VERITAS Cluster Server (tm) cluster. This information can be useful when troubleshooting potential interconnect issues on the heartbeat network between cluster nodes.
The LLT configuration file /etc/llttab
describes the systems in the cluster and private network links among them. LLT runs in layer 2 of the network stack.
Typical output
Without applying any options lltstat
outputs link statistics
root@solnod1# lltstat LLT statistics: 988803 Snd data packets 495884 Rcv data packets 11400 Snd retransmit data 687 Rcv out of window 28824991 Snd connect packets 0 Rcv duplicates 1651 Snd independent ACKs 104 Rcv out of sequence 494559 Snd piggyback ACKs 0 Rcv datagrams dropped 0 Snd indenepdent NACKs 1481482 Snd loopback packets 0 Snd Piggybank NACKs LLT errors: 0 Rcv not connected 4 Snd not connected 0 Rcv unconfigured 0 Snd no buffer 0 Rcv bad dest address 0 Snd error drops 0 Rcv bad source address 625 Snd no links up 0 Rcv bad generation 0 0 Rcv malformed packet 0
Note: There are three columns of data, followed by another three columns next to it
- — contains the number of packets sent or received
- — denotes either Snd (Data Sent) or Rcv (Data Received)
- — provides a short description of each statistic (see below for detailed description)
List Peer Systems
Whilst, the lltstat
command is used to interrogate the LLT protocol. The -n
option displays the current status of the peer systems.
root@solnod1# lltstat -n LLT node information: Node State Links * 0 solnod1 OPEN 3 1 solnod2 OPEN 3
root@solnod2# lltstat -n LLT node information: Node State Links 0 solnod1 OPEN 3 * 1 solnod2 OPEN 3
Note Each node has three links and that each node is in the open state. The asterisk (*) denotes the node on which you typed the command.
The -n
option may be used with the -v
or -vv
(verbose and very-verbose options, respectively) to display link information and MAC addresses.
root@solnod1# lltstat -nv LLT node information: Node State Links * 0 solnod1 OPEN 3 1 solnod2 OPEN 3 2 CONNWAIT 0 3 CONNWAIT 0 : : {snip} : 31 CONNWAIT 0 root@solnod1# lltstat -nvv LLT node information: Node State Link Status Address * 0 solnod1 OPEN ce0 UP 08:00:20:B4:C5:68 ce7 UP 08:00:20:B4:C4:01 ce3 UP 08:00:20:B6:B5:D9 1 solnod2 OPEN ce0 UP 08:00:20:B4:9E:36 ce7 UP 08:00:20:B4:A2:6D ce3 DOWN 2 CONNWAIT ce0 DOWN ce7 DOWN ce3 DOWN 3 CONNWAIT ce0 DOWN ce7 DOWN ce3 DOWN : : {snip} : 31 CONNWAIT ce0 DOWN ce7 DOWN ce3 DOWN
Note: The output lists all possible 32 nodes of the cluster. It reports on the two nodes in the cluster, solnod1
and solnod2
, plus non-existent nodes.
For each correctly configured node, the information should show a state of open, a status for each link of up, and an address for each link. However, the output in the example shows that for the solnod2
node the private network connection is possibly broken or the information in the /etc/llttab
file may be incorrect.
LLT port status
To obtain information about the ports open for LLT, type lltstat -p
on any node. For example:
solnod2# lltstat -p LLT port information: Port Usage Cookie 0 gab 0x0 opens: 0 2 3 4 5 6 7 8 9 10 11 ... 60 61 62 63 connects: 0 1 7 gab 0x7 opens: 0 2 3 4 5 6 7 8 9 10 11 ... 60 61 62 63 connects: 0 1 31 gab 0x1F opens: 0 2 3 4 5 6 7 8 9 10 11 ... 60 61 62 63 connects: 0 1
List Kernel Tunables
Use the lltstat -t
on any node to list the LLT kernel tunables. For example:
solnod1# lltstat -t LLT kernel tunables: maxnids: 32 maxports: 32 nominpad: 0 rthreadpri: 98 basetimer: 50000
Note: The basetimer
is a timeout value of 50ms (50000 microseconds).
List Link information
The lltstat -l
lists information about each configured LLT link. For example:
solnod1# lltstat -l LLT link information: Link Tag State Type Pri SAP MTU Addrlen Xmit Recv Err LateHB Broadcast 0 ce0 on ether hipri 0xCAFE 1500 6 4805041 96371727 0 109 FF:FF:FF:FF:FF:FF 1 ce7 on ether hipri 0xCAFE 1500 6 4805041 96371727 0 114 FF:FF:FF:FF:FF:FF 2 ce3 on ether hipri 0xCAFE 1500 6 4806098 76384154 0 122 FF:FF:FF:FF:FF:FF
Note: link 0 is ce0, link 1 is ce7 and link 2 is ce3. This should also correspond with what is in /etc/llttab
.
List Configuration Directives
The lltstat -c
to display LLT configuration directives. For example:
solnod1# lltstat -c LLT configuration information: node: 1 name: sunclu2 cluster: 102 version: 1.1 nodes: 0 - 31 max nodes: 32 max ports: 32 links: 3 mtu: 1460 max sdu: 66560 broadcast HB: 1 warning level: 20
If you simply want to display the cluster ID, you can pipe the output to grep. For example:
# lltstat -c | grep cluster cluster: 102
lltstat
link statistics explained
The table below list expands on the lltstat
link statistics
Short Name | Description |
---|---|
Snd independent ACKs | number of ACK packets sent |
Snd connect packets | number of CR/CC/DR/DC packets sent |
Snd stream flow drops | xmit drops, can't put next to driver |
Snd loopback packets | xmit loopback packets |
Snd independent NACKs | number of NAK packets sent |
Snd no buffer | no mem for bufs to xmit |
Snd not connected | node is not in connected state |
Snd no links up | no link to xmit packets |
Snd piggyback ACKs | number of piggybacked ACKs sent |
Snd piggyback NACKs | number of piggybacked NAKs sent |
Snd retransmit data | number of re-transmissions |
Snd data packets | number of user pkts sent |
Rcv bad generation | connection generation num mismatch |
Rcv bad dest address | destid doesn't match our nid |
Rcv datagrams dropped | received datagrams dropped |
Rcv bad DLPI primitive | received unexpected DLPI primitive |
Rcv DLPI error | received DLPI error_ack primitive |
Rcv duplicates | received duplicate packets |
Rcv malformed packet | received malformed packet |
Rcv misaligned data | Unaligned packets received |
Rcv multiblock Data | Data received in multiple mblks |
Rcv no buffer | no mem for bufs to recv |
Rcv not connected | node is not in connected state |
Rcv out of sequence | out-of-sequence packets |
Rcv out of window | out-of-window |
Rcv bad dest SAP | received SAP is not ours |
Rcv bad source address | src node id is out of range |
Rcv bad STREAM primitive | received unexpected stream primitive |
Rcv unconfigured | src node is not configured |
Rcv data packets | number of user pkts received |