Understanding LLTSTAT output
The LLTSTAT command line tool can be used to display statistics of the Low Latency Transport (LLT) protocol that is used for the heartbeats of a VERITAS Cluster Server (tm) cluster. This information can be useful when troubleshooting potential interconnect issues on the heartbeat network between cluster nodes.
The LLT configuration file /etc/llttab describes the systems in the cluster and private network links among them. LLT runs in layer 2 of the network stack.
Typical output
Without applying any options lltstat outputs link statistics
root@solnod1# lltstat
LLT statistics:
988803 Snd data packets 495884 Rcv data packets
11400 Snd retransmit data 687 Rcv out of window
28824991 Snd connect packets 0 Rcv duplicates
1651 Snd independent ACKs 104 Rcv out of sequence
494559 Snd piggyback ACKs 0 Rcv datagrams dropped
0 Snd indenepdent NACKs 1481482 Snd loopback packets
0 Snd Piggybank NACKs
LLT errors:
0 Rcv not connected 4 Snd not connected
0 Rcv unconfigured 0 Snd no buffer
0 Rcv bad dest address 0 Snd error drops
0 Rcv bad source address 625 Snd no links up
0 Rcv bad generation 0
0 Rcv malformed packet 0
Note: There are three columns of data, followed by another three columns next to it
- — contains the number of packets sent or received
- — denotes either Snd (Data Sent) or Rcv (Data Received)
- — provides a short description of each statistic (see below for detailed description)
List Peer Systems
Whilst, the lltstat command is used to interrogate the LLT protocol. The -n option displays the current status of the peer systems.
root@solnod1# lltstat -n
LLT node information:
Node State Links
* 0 solnod1 OPEN 3
1 solnod2 OPEN 3
root@solnod2# lltstat -n
LLT node information:
Node State Links
0 solnod1 OPEN 3
* 1 solnod2 OPEN 3
Note Each node has three links and that each node is in the open state. The asterisk (*) denotes the node on which you typed the command.
The -n option may be used with the -v or -vv (verbose and very-verbose options, respectively) to display link information and MAC addresses.
root@solnod1# lltstat -nv
LLT node information:
Node State Links
* 0 solnod1 OPEN 3
1 solnod2 OPEN 3
2 CONNWAIT 0
3 CONNWAIT 0
:
: {snip}
:
31 CONNWAIT 0
root@solnod1# lltstat -nvv
LLT node information:
Node State Link Status Address
* 0 solnod1 OPEN
ce0 UP 08:00:20:B4:C5:68
ce7 UP 08:00:20:B4:C4:01
ce3 UP 08:00:20:B6:B5:D9
1 solnod2 OPEN
ce0 UP 08:00:20:B4:9E:36
ce7 UP 08:00:20:B4:A2:6D
ce3 DOWN
2 CONNWAIT
ce0 DOWN
ce7 DOWN
ce3 DOWN
3 CONNWAIT
ce0 DOWN
ce7 DOWN
ce3 DOWN
:
: {snip}
:
31 CONNWAIT
ce0 DOWN
ce7 DOWN
ce3 DOWN
Note: The output lists all possible 32 nodes of the cluster. It reports on the two nodes in the cluster, solnod1 and solnod2, plus non-existent nodes.
For each correctly configured node, the information should show a state of open, a status for each link of up, and an address for each link. However, the output in the example shows that for the solnod2 node the private network connection is possibly broken or the information in the /etc/llttab file may be incorrect.
LLT port status
To obtain information about the ports open for LLT, type lltstat -p on any node. For example:
solnod2# lltstat -p
LLT port information:
Port Usage Cookie
0 gab 0x0
opens: 0 2 3 4 5 6 7 8 9 10 11 ... 60 61 62 63
connects: 0 1
7 gab 0x7
opens: 0 2 3 4 5 6 7 8 9 10 11 ... 60 61 62 63
connects: 0 1
31 gab 0x1F
opens: 0 2 3 4 5 6 7 8 9 10 11 ... 60 61 62 63
connects: 0 1
List Kernel Tunables
Use the lltstat -t on any node to list the LLT kernel tunables. For example:
solnod1# lltstat -t
LLT kernel tunables:
maxnids: 32
maxports: 32
nominpad: 0
rthreadpri: 98
basetimer: 50000
Note: The basetimer is a timeout value of 50ms (50000 microseconds).
List Link information
The lltstat -l lists information about each configured LLT link. For example:
solnod1# lltstat -l
LLT link information:
Link Tag State Type Pri SAP MTU Addrlen
Xmit Recv Err LateHB
Broadcast
0 ce0 on ether hipri 0xCAFE 1500 6
4805041 96371727 0 109
FF:FF:FF:FF:FF:FF
1 ce7 on ether hipri 0xCAFE 1500 6
4805041 96371727 0 114
FF:FF:FF:FF:FF:FF
2 ce3 on ether hipri 0xCAFE 1500 6
4806098 76384154 0 122
FF:FF:FF:FF:FF:FF
Note: link 0 is ce0, link 1 is ce7 and link 2 is ce3. This should also correspond with what is in /etc/llttab.
List Configuration Directives
The lltstat -c to display LLT configuration directives. For example:
solnod1# lltstat -c
LLT configuration information:
node: 1
name: sunclu2
cluster: 102
version: 1.1
nodes: 0 - 31
max nodes: 32
max ports: 32
links: 3
mtu: 1460
max sdu: 66560
broadcast HB: 1
warning level: 20
If you simply want to display the cluster ID, you can pipe the output to grep. For example:
# lltstat -c | grep cluster
cluster: 102
lltstat link statistics explained
The table below list expands on the lltstat link statistics
| Short Name | Description |
|---|---|
| Snd independent ACKs | number of ACK packets sent |
| Snd connect packets | number of CR/CC/DR/DC packets sent |
| Snd stream flow drops | xmit drops, can't put next to driver |
| Snd loopback packets | xmit loopback packets |
| Snd independent NACKs | number of NAK packets sent |
| Snd no buffer | no mem for bufs to xmit |
| Snd not connected | node is not in connected state |
| Snd no links up | no link to xmit packets |
| Snd piggyback ACKs | number of piggybacked ACKs sent |
| Snd piggyback NACKs | number of piggybacked NAKs sent |
| Snd retransmit data | number of re-transmissions |
| Snd data packets | number of user pkts sent |
| Rcv bad generation | connection generation num mismatch |
| Rcv bad dest address | destid doesn't match our nid |
| Rcv datagrams dropped | received datagrams dropped |
| Rcv bad DLPI primitive | received unexpected DLPI primitive |
| Rcv DLPI error | received DLPI error_ack primitive |
| Rcv duplicates | received duplicate packets |
| Rcv malformed packet | received malformed packet |
| Rcv misaligned data | Unaligned packets received |
| Rcv multiblock Data | Data received in multiple mblks |
| Rcv no buffer | no mem for bufs to recv |
| Rcv not connected | node is not in connected state |
| Rcv out of sequence | out-of-sequence packets |
| Rcv out of window | out-of-window |
| Rcv bad dest SAP | received SAP is not ours |
| Rcv bad source address | src node id is out of range |
| Rcv bad STREAM primitive | received unexpected stream primitive |
| Rcv unconfigured | src node is not configured |
| Rcv data packets | number of user pkts received |