04 Sep 09 Q4 Crash Dump Analysis to Analyze System Dump Files

What follows is a document I found on the forums. It can also be found on the docs.hp.com site, but this is a paraphrase, with some extra commentary.

1) You need to have foresight. Before you have a crash you must enable your system to save crash dumps.

2) vi /etc/rc.config.d/savecrash — set the first parameter to 1. Now when your system crashes, and some day it probably will you can perform q4 analysis and send the results to HP. I think this document originated within HP. I have one written somewhere on the forums, but his one is better.

USING Q4 TO ANALYZE SYSTEM DUMP FILES

————————————-

When a 11.X HP-UX system crashes, it saves a snapshot of RAM in swap and during the reboot, copies it into /var/adm/crash. Because these files are binary, a utility called “q4” is used to analyze them and create readable text from which the response center can determine the failure cause.

============================ STEP 1 ===========================

Dumps are normally saved to /var/adm/crash.

Verify you have a dump to analyze by doing:

# ll /var/adm/crash/cr*

You may see:

/var/adm/crash/crash.0/INDEX

/var/adm/crash/crash.0/vmunix.gz

/var/adm/crash/crash.0/image.0.1.gz

/var/adm/crash/crash.0/image.0.2.gz

/var/adm/crash/crash.0/image.0.3.gz

/var/adm/crash/crash.0/image.0.4.gz

^ your suffix may vary

The INDEX file contains and the /etc/shutdownlog contains the “panic” statement.

============================ STEP 2 ===========================

The following commands must all be run from the dump directory:

cd to the dump directory ie: cd /var/adm/crash/crash.0

^^^^ ^

your dump dir.

# /usr/contrib/bin/gunzip vmunix.gz

(uncompresses the kernel file – may already be done)

# q4prep -p

(ignore the error if this was previously done)

Now type:

# q4 -p .

^ Notice this ‘dot’

This will put you at the q4 utility prompt: q4>

The next command will get you a “fingerprint” of what was going on on the system at the time of the failure.

If you are working with an HP RCE at this time, type the following line and read the results to him:

trace event 0

Otherwise, simply type this next line and continue.

trace event 0 > trace

At the prompt type: include analyze.pl

\_letter “el”

At the next prompt type: run Analyze AU >> ana.out

At the next prompt type: exit

============================ STEP 3===========================

Generate a patch list:

# swlist -l product PH\* > patch_list

Using the CALL ID as the subject, email patch_list, ana.out and possibly the trace file and what.out to : hpcu@atl.hp.com

NOTE: Max 3MB email size

To speed future calls of this nature, open a call with the Response Center and inform them that you will send email with the call ID as the subject. Then send the ana.out and patch_list file to the email address listed above.

Tags: crash dump, forums.itrc.hp.com, HP-UX, q4