Solaris Crash Dumps
If a SunOS/Solaris system crashes (also known as a "panic") it is possible to write the entire contents of memory to disk for later analysis. It is highly recommended to enable so called crash dumps on the SunOS/Solaris servers.
Note: Care must be taken that enough disk space is available (e.g. on large db servers with 500MB memory, it may not be possible to dump to swap space).
- For Solaris 1 (SunOS 4.x) edit
/etc/rc.localand uncomment the following lines:
# # Enable savecore (default is disabled) # mkdir -p /var/crash/`hostname` echo -n 'checking for crash dump... ' intr savecore /var/crash/`hostname` echo ''
- For Solaris 2 edit the file
/etc/init.d/sysetupand uncomment the following lines:
## ## Enable savecore (default is disabled) ## if [ ! -d /var/crash/`uname -n` ] then mkdir -p /var/crash/`uname -n` fi echo 'checking for crash dump...\c ' savecore /var/crash/`uname -n` echo ''
- All crash dumps could have highly confidential information since they contain all application memory space at the time of the crash. I highly recommend to add the following line to the above files, to prevent unauthorised access to the dumps.
chown -R root.staff /var/crash chmod -R 600 /var/crash
- If the file
minfreeexits in the crash directory, the number in this file specifies how many kilobytes of space must remain free on this filesystem once
- Dump to a special (i.e. not the swap) device is possible. On Solaris 1, add the line to the kernel configuration file (assuming you want to use device sd1b) and rebuild the kernel (see
config vmunix swap on sd1bOn Solaris 2, it's a bit trickier,
adbmust be used.
- If the several system crashes are expected, compress previous dumps. They often compress to only 5% of the original size. The same goes for dumps which are archived.
- Crash dumps MUST be analysed on the same OS version and architecture as they were created (with
Initial Crash Dump analysis
The following commands can be used to analyse what was going on in the system before the panic occurred.
Using CRASH:/etc/crash -d vmcore.0 -n vmunix.0/usr/sbin/crash -d vmcore.0 -n vmunix.0
|Description||Solaris 1 (SunOS 4)||Solaris 2.x|
|What OS is this?||strings vmcore.0 | grep SunOS||strings vmcore.0 | grep SunOS|
|What host is this?||strings vmcore.0 | grep machine||strings vmcore.0 | grep machine|
|What processes were running?||ps -laxk vmunix.0 vmcore.0||use crash (see below)|
|Show system tables||pstat -T vmunix.0 vmcore.0|
|Show network stats||netstat vmunix.0 vmcore.0||netstat -d unix.0 vmcore.0|
|Show NFS stats||nfsstat -n vmunix.0 vmcore.0||nfsstat -n unix.0 vmcore.0|
|Show arp table||arp -c vmunix.0 vmcore.0||arp -a unix.0 vmcore.0|
|Show IPC stuff||ipcs -a -N vmunix.0 -C vmcore.0||ipcs -a -N unix.0 -C vmcore.0|
|crash help||> help||> help|
|Help on "p" command||> help p||> help p|
|Show processes||> p -e||> p -e|
|Lots of process details||> p -l|
|crash details||> status|
|Quit crash||> q||> q|
|Using the ADB debugger:||adb -k vmunix.0 vmcore.0||adb -k unix.0 vmcore.0|
|What was the panic message?||*panicstr/s||*panicstr/s|
|Display system messages||msgbuf+10/s||msgbuf+14s|
|Recent message buffer (ring)||$<msgbuf||$<msgbuf|
|C stack traceback
(not always right!)
|What is root device?||rootfs$<bootobj|
|What is swap device?||swapfile$<bootobj
|Show IPC stuff||ipcaccess/10i|
|Quit adb||CTRL-D or $q||CTRL-D or $q|
|Access a live kernel):||adb -k /vmunix /dev/mem||adb -k /dev/ksyms /dev/mem|
adb macros are located in /usr/lib/adb (Solaris 1) or /usr/kvm/lib/adb (Solaris 2).