Troubleshooting Solaris memory usage
With Solaris 9 and later, comes an extremly nice mdb command for checking how much memory is used in kernel, and how much is used in user space.
# echo "::memstat" | mdb -k Page Summary Pages MB %Tot ------------ ---------------- ---------------- ---- Kernel 309761 2420 15% ZFS File Data 1368646 10692 66% Anon 135201 1056 7% Exec and libs 5714 44 0% Page cache 93686 731 5% Free (cachelist) 10078 78 0% Free (freelist) 135334 1057 7% Total 2058420 16081 Physical 2036844 15912
# prstat -s size
# prstat -s rrs
The above two commands will display prcesses using most USER memory, (the sort rss sorts by physical memory, size sorted by virtual memory used)
# prstat -s rss
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
833 root 175M 102M sleep 59 0 0:03:30 0.1% java/57
911 noaccess 206M 100M sleep 59 0 0:00:59 0.0% java/19
2885 root 85M 65M sleep 59 0 0:12:07 1.0% nsrjobd/18
2865 root 67M 41M sleep 49 0 0:05:03 0.6% nsrd/4
4061 root 53M 28M cpu5 59 -15 0:16:20 3.0% nsrmmd/2
:
:
Total: 144 processes, 1016 lwps, load averages: 2.12, 2.12, 2.44
# prstat -s size
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
911 noaccess 206M 100M cpu7 59 0 0:00:59 0.0% java/19
833 root 175M 102M sleep 59 0 0:03:30 0.1% java/57
2885 root 85M 65M sleep 59 0 0:12:08 0.6% nsrjobd/18
2865 root 67M 41M sleep 59 0 0:05:03 0.2% nsrd/4
1443 root 66M 36M sleep 44 5 0:16:41 0.3% eba_collector/8
:
:
Total: 144 processes, 1014 lwps, load averages: 2.13, 2.12, 2.43
Shared memory and semaphores
# ipcs -a IPC status from as of Monday, 28 May 2010 17:43:44 BST T ID KEY MODE OWNER GROUP CREATOR CGROUP CBYTES QNUM QBYTES LSPID LRPID STIME RTIME CTIME Message Queues: T ID KEY MODE OWNER GROUP CREATOR CGROUP NATTCH SEGSZ CPID LPID ATIME DTIME CTIME Shared Memory: m 9 0 --rw-rw-rw- root root root root 1 5914624 4063 4105 17:00:54 17:33:39 16:15:02 m 8 0 --rw-rw-rw- root root root root 2 5914624 4057 4061 17:37:11 17:35:39 16:05:52 m 7 0 --rw-rw-rw- root root root root 1 5914624 4061 4064 16:59:12 17:22:29 14:25:01 m 6 0 --rw-rw-rw- root root root root 1 5914624 4059 1405 13:55:16 13:55:33 13:55:16 m 5 0 --rw-rw-rw- root root root root 1 5914624 4105 4105 12:47:40 no-entry 12:47:40 m 4 0 --rw-rw-rw- root root root root 1 5914624 4064 4065 12:20:38 13:59:26 12:20:01 m 3 0 --rw-rw-rw- root root root root 1 5914624 4062 4063 12:20:53 16:05:25 12:19:00 m 2 0 --rw-rw-rw- root root root root 1 5914624 4060 19292 12:18:43 12:18:48 12:18:43 m 1 0 --rw-rw-rw- root root root root 1 5914624 4053 16563 12:00:02 13:55:15 12:00:02 m 0 0x7a00aca0 --rw-rw-rw- zabbix other zabbix other 6 1099904 1325 1325 10:12:48 no-entry 10:12:48 T ID KEY MODE OWNER GROUP CREATOR CGROUP NSEMS OTIME CTIME Semaphores:
Listing user memory usage
# pmap -x /proc/* > /var/tmp/pmap-x short list of total usage of these processes # egrep “[0-9]:|^total” /var/tmp/pmap-x 1: /sbin/init total Kb 2336 2080 128 - 1006: rlogin cores4 total Kb 2216 1696 80 - 1007: rlogin cores4 total Kb 2216 1696 104 - 115: /usr/sbin/nscd total Kb 4208 3784 1704 - — snip —
memory used by /tmp
# df -kl /tmp Filesystem kbytes used avail capacity Mounted on swap 1024000 80 1023920 1% /tmp
10 bigest files/dirs under a given file directory
# du -akd /tmp | sort -n | tail -10 8 /tmp/hsperfdata_root 8 /tmp/jnlpgenerator-8 8 /tmp/jnlpgenerator-8-1 8 /tmp/jnlpgenerator-8-2 8 /tmp/nscopy.tmp 8 /tmp/nsmail.eml 8 /tmp/speckeysd.lock 16 /tmp/tt 48 /tmp/lucreate.man 800 /tmp/dtdbcache_:0 1088 /tmp
Using vmstat to view overall memory usage on a server
# vmstat -p 3
memory page executable anonymous filesystem
swap free re mf fr de sr epi epo epf api apo apf fpi fpo fpf
18149352 4080496 51 270 2 0 17 11 0 0 0 0 0 29 2 2
15384792 1086168 1 29 0 0 0 8 0 0 0 0 0 135 0 0
15384280 1085768 72 297 0 0 0 0 0 0 0 0 0 0 0 0
swap usage
# swap -l swapfile dev swaplo blocks free /dev/md/dsk/d20 85,20 16 32787056 32787056 # swap -s total: 1124728k bytes allocated + 558824k reserved = 1683552k used, 15399632k available
print out all kernel statistics in a parse'able format
# kstat -p
kernel memory statistics
# kstat -p -c kmem_cache # kstat -p -m vmem # kstat -p -c vmem
alternatively to kstat you can get kernel memory usage with kmastat prints kmastat buffers
# echo “::kmastat” | mdb -k
cache buf buf buf memory alloc alloc
name size in use total in use succeed fail
------------------------- ------ ------ ------ --------- --------- -----
kmem_magazine_1 16 1781 23368 376832 1360984 0
kmem_magazine_3 32 26120 26416 851968 144009 0
kmem_magazine_7 64 20265 24511 1581056 373845 0
:
:
msqids 0 128 0 0 0
semids 0 128 0 0 0
------------------------- --------- ---------- --------- --------- -----