Simple instructions for using the AIX trace facility

AIX facilities used

Five aix commands are used:

trace
trcon
trcoff
trcstop
trcrpt

These are described in AIX Commands Reference, Volume 5, but hopefully you won't have to dig into that.

Scripts to download

I've provided wrappers for the trace and trcrpt commands since there are various command-line parameters to specify.

Setup instructions

edit atrace and atrcrpt and ensure that names of files for raw and formatted trace are appropriate
Please see the comments in the scripts about 4.3.3 ML 10 being broken for trcrpt, such that the default file name needs to be used. You may find that specifying non-default filenames does not have the desired effect.
make atrace and atrcrpt executable via chmod

Data collection

./atrace                 (this is my wrapper for the trace command)
trcon
(at this point we're collecting the trace; wait for a bit of time to
trace whatever the failure is)
trcoff
trcstop
./atrcrpt                (this is my wrapper for formatting the report)

After running atrcrpt, the formatted report will be in file /tmp/fmt.

Sample section of formatted trace

Note that failing system calls generally show "error Esomething" in the race, as highlighted below.

The second column is the process id and the third column is the thread id. Once you see something of interest in the trace, you may want to use grep to pull out all records for that process id, since in general the trace is interleaved with the activity of all the processes in the system.

101 14690    19239              statx LR = D0174110
107 14690    19239                      lookuppn: /usr/HTTPServer/htdocs/en_US/manual/ibm/index.htmlxxxxxxxxxxx
107 14690    19239                      lookuppn: file not found
104 14690    19239              return from statx. error ENOENT [79 usec]
101 14690    19239              statx LR = D0174110
107 14690    19239                      lookuppn: /usr/HTTPServer/htdocs/en_US/manual/ibm
104 14690    19239              return from statx [36 usec]