Troubleshooting Solaris memory usage

With Solaris 9 and later, comes an extremly nice mdb command for checking how much memory is used in kernel, and how much is used in user space.

# echo "::memstat" | mdb -k
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                     309761              2420   15%
ZFS File Data             1368646             10692   66%
Anon                       135201              1056    7%
Exec and libs                5714                44    0%
Page cache                  93686               731    5%
Free (cachelist)            10078                78    0%
Free (freelist)            135334              1057    7%

Total                     2058420             16081
Physical                  2036844             15912
# prstat -s size
# prstat -s rrs

The above two commands will display prcesses using most USER memory, (the sort rss sorts by physical memory, size sorted by virtual memory used)

# prstat -s rss
   PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP
   833 root      175M  102M sleep   59    0   0:03:30 0.1% java/57
   911 noaccess  206M  100M sleep   59    0   0:00:59 0.0% java/19
  2885 root       85M   65M sleep   59    0   0:12:07 1.0% nsrjobd/18
  2865 root       67M   41M sleep   49    0   0:05:03 0.6% nsrd/4
  4061 root       53M   28M cpu5    59  -15   0:16:20 3.0% nsrmmd/2
            :
            :
Total: 144 processes, 1016 lwps, load averages: 2.12, 2.12, 2.44
# prstat -s size
   PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP
   911 noaccess  206M  100M cpu7    59    0   0:00:59 0.0% java/19
   833 root      175M  102M sleep   59    0   0:03:30 0.1% java/57
  2885 root       85M   65M sleep   59    0   0:12:08 0.6% nsrjobd/18
  2865 root       67M   41M sleep   59    0   0:05:03 0.2% nsrd/4
  1443 root       66M   36M sleep   44    5   0:16:41 0.3% eba_collector/8
            :
            :
Total: 144 processes, 1014 lwps, load averages: 2.13, 2.12, 2.43

Shared memory and semaphores

# ipcs -a
IPC status from as of Monday, 28 May 2010 17:43:44 BST
T         ID      KEY        MODE        OWNER    GROUP  CREATOR   CGROUP CBYTES  QNUM QBYTES LSPID LRPID   STIME    RTIME    CTIME
Message Queues:
T         ID      KEY        MODE        OWNER    GROUP  CREATOR   CGROUP NATTCH      SEGSZ  CPID  LPID   ATIME    DTIME    CTIME
Shared Memory:
m          9   0          --rw-rw-rw-     root     root     root     root      1    5914624  4063  4105 17:00:54 17:33:39 16:15:02
m          8   0          --rw-rw-rw-     root     root     root     root      2    5914624  4057  4061 17:37:11 17:35:39 16:05:52
m          7   0          --rw-rw-rw-     root     root     root     root      1    5914624  4061  4064 16:59:12 17:22:29 14:25:01
m          6   0          --rw-rw-rw-     root     root     root     root      1    5914624  4059  1405 13:55:16 13:55:33 13:55:16
m          5   0          --rw-rw-rw-     root     root     root     root      1    5914624  4105  4105 12:47:40 no-entry 12:47:40
m          4   0          --rw-rw-rw-     root     root     root     root      1    5914624  4064  4065 12:20:38 13:59:26 12:20:01
m          3   0          --rw-rw-rw-     root     root     root     root      1    5914624  4062  4063 12:20:53 16:05:25 12:19:00
m          2   0          --rw-rw-rw-     root     root     root     root      1    5914624  4060 19292 12:18:43 12:18:48 12:18:43
m          1   0          --rw-rw-rw-     root     root     root     root      1    5914624  4053 16563 12:00:02 13:55:15 12:00:02
m          0   0x7a00aca0 --rw-rw-rw-   zabbix    other   zabbix    other      6    1099904  1325  1325 10:12:48 no-entry 10:12:48
T         ID      KEY        MODE        OWNER    GROUP  CREATOR   CGROUP NSEMS   OTIME    CTIME
Semaphores:

Listing user memory usage

# pmap -x /proc/* > /var/tmp/pmap-x 
short list of total usage of these processes
 
# egrep “[0-9]:|^total” /var/tmp/pmap-x 
1:   /sbin/init
 total Kb 2336 2080  128 -
 1006:  rlogin cores4
 total Kb 2216 1696    80 -
 1007:  rlogin cores4
 total Kb 2216 1696  104 -
 115:  /usr/sbin/nscd
 total Kb 4208 3784 1704 -
 — snip —

memory used by /tmp

# df -kl /tmp
Filesystem            kbytes    used   avail capacity  Mounted on
swap                 1024000      80 1023920     1%    /tmp

10 bigest files/dirs under a given file directory

# du -akd /tmp | sort -n | tail -10
8  /tmp/hsperfdata_root
8 /tmp/jnlpgenerator-8
8 /tmp/jnlpgenerator-8-1
8 /tmp/jnlpgenerator-8-2
8 /tmp/nscopy.tmp
8 /tmp/nsmail.eml
8 /tmp/speckeysd.lock
16 /tmp/tt
48 /tmp/lucreate.man
800 /tmp/dtdbcache_:0
1088 /tmp

Using vmstat to view overall memory usage on a server

# vmstat -p 3
     memory           page          executable      anonymous      filesystem
   swap  free  re  mf  fr  de  sr  epi  epo  epf  api  apo  apf  fpi  fpo  fpf
 18149352 4080496 51 270 2  0  17   11    0    0    0    0    0   29    2    2
 15384792 1086168 1 29  0   0   0    8    0    0    0    0    0  135    0    0
 15384280 1085768 72 297 0  0   0    0    0    0    0    0    0    0    0    0

swap usage

# swap -l
swapfile             dev  swaplo blocks   free
/dev/md/dsk/d20     85,20     16 32787056 32787056

# swap -s
total: 1124728k bytes allocated + 558824k reserved = 1683552k used, 15399632k available

print out all kernel statistics in a parse'able format

# kstat -p

kernel memory statistics

# kstat -p -c kmem_cache
# kstat -p -m vmem
# kstat -p -c vmem

alternatively to kstat you can get kernel memory usage with kmastat prints kmastat buffers

# echo “::kmastat” | mdb -k
cache                        buf    buf    buf    memory     alloc alloc
name                        size in use  total    in use   succeed  fail
------------------------- ------ ------ ------ --------- --------- -----
kmem_magazine_1               16   1781  23368    376832   1360984     0
kmem_magazine_3               32  26120  26416    851968    144009     0
kmem_magazine_7               64  20265  24511   1581056    373845     0
            :
            :
msqids                            0        128         0         0     0
semids                            0        128         0         0     0
------------------------- --------- ---------- --------- --------- -----