Utility Script to format OpenPegasus trace output

20 March 2014
Karl Schopmeyer

OVERVIEW
formattrace is a script/awkscript tool to provide some formatting to OpenPegasus
trace files  and make the resulting trace file more readable and usable.

It performs the following:

1. Consolidate the multiple trace files from a trace with OOP enabled into
a single file ordered by date/time. Since OpenPegasus produces a trace
file for each provider agent (i.e. separate provider process) when providers
are run out-of-process, this results in a number of files for a single trace
where it is difficult to view the flow through the system. formattrace
consolidates all of the files into a single trace file sorted by the
time of each log entry.

2. Allow selective start of the trace output based on the first instance
of a particular trace component in the output.  This eliminates the
extraneous trace output that occurs during OpenPegasus startup and is normally
not of any use in debugging issues.  Thus, for example, the default
is to start the trace only when the first operation request is received
(the first trace line with trace component = XmlIO)

3. Modify the time field on trace logs so that they start at 0 seconds for
the first trace line in the output.  This makes it easier to see time
differences in the trace output.  Thus, the relative time for the
trace is relative to the first trace entry output by formattrace.

4. Allow marking selected trace lines (first character set to +) to
identify trace lines that might be important.

5. Modify the process and thread identifiers to simple names in place of
numbers (ex. P01 for the first process in the trace) making it easier to
track the trace of particular threads and processes.

Today this works only for Unix/Linux systems that have both bash shell
and awk available.

HOW TO USE formattrace

1. Install all of the components  (traceformat script and the *.awk scripts)
in a directory in the path.

2. Insure that PEGASUS_HOME variable exists. traceformat locates the original
trace files through the $PEGASUS_HOME/trace

Generally the format for the command line script is:

    formattrace <command> <options>

    where the commands are today:
        format
	clean

3. Execute the format with:

    formattrace format

This produces two output files in the current directory:
   1. File with date time in it and suffix of text.  This is summary information
      special warnings, etc. (ex. CIM_Tr_077_091213.SummaryInfo.txt)
   2. Same fileName but suffix of fmtd.  This is the consolidated formatted
      trace output (ex. CIMTr_078_091213.fmtd).

      Note that intermediate files are produced for the consolidation and
      sort but are deleted by the script

4. The resulting CIMTr_ddd_hhmmss.fmtd file can be viewed and manipulated
by any text editor.

4. To clean up produced formatted traces, there are options to:
   1. remove particular trace files (formattrace remove).  Will show existing
      formatted files and you can select 1 to remove.
   2. clean out all trace output files (formattrace clean or formattrace format -c)


TODO List

This is a prototype and early version. We wrote it with a scripting language
because we needed something for a couple of fairly complex bugs and also
understood that Pegasus traces can be complex to work with. We are looking for suggestions to
make this more usable.

TODOs on the agenda already include:

1. Extend the options to allow more parameters that would define what is
included in the *.fmtd file and also to further filter that file.

1. Find ways to identify all of the trace elements for a particular operation
so that a complete operation can be traced through the system.

2. Make the selection of marks more flexible by extending the script so that
they could be set or unset in the already created output (*.fmtd) file.

3. Remove trace elements from the end of the trace (ex. the shutdown) so that
the resulting trace is just the operations information.

4. Remove or hide selected trace component elements.  This would allow the
user to just view the important issues.

5. Add option to allow manual selection of the source trace files in stead
of requiring PEGASUS_HOME