Return to formattrace CVS log

Up to [Pegasus] / pegasus_unsupported / scripts / formattrace

BUG#:9999
TITLE: Add scripts for chotesting and update the formattrace scripts

DESCRIPTION: These are unsupported test extensions for use in pegasus
development and testing.  Each has help and readme files.

#!/bin/bash
##
## fix a set of cimserver.xxx.trc files into a single file and
## and substitute data to Make the result more readable
##
## The --posix or corresponding -re-interval options are required to include
## the regex expression interval logic ( i.e. [1-9]{5}).
# Discover where the shell script resides assuming other scripts are on
## same directory

AWK="awk --posix"

## Goal will be to use mawk if possible because it is about 3x faster 
## than gnu awk.  However, there are a number of differences and right
## it fails to execute the two awk scripts.  KS 22 April 2014
##AWK=mawk


OUTFILEPREFIX=CIMTr
OUTFILE=$OUTFILEPREFIX$(date +_%j_%H%M%S)

PROGNAME=`type $0 | awk '{print $3}'`  # search for this script on path
echo $PROGNAME
PROGDIR=`dirname $PROGNAME`            # extract directory of program

######################################################################
## Functions
#####################################################################
# usage and show directories if no parameter
function USAGE()
{
cat << EOF
Usage: `basename $0` command keyword <parameters>
  where commands are:
    format             format a set of trace files from the directory define
                       by $PEGASUS_HOME/trace and put result into current
                       directory
    remove |rm         select and remove a formatted result
    clean              remove all existing trace output files
    fix                fixup and existing trace by setting more marks
    help               display overview and help info.
  and options are:
    -d|--debug         set debug flags for the scripts that format. Diagnostic
                       for these scripts.
    -c|--clean         clean. Corresponds to clean command but allows 
                       cleaning all old traces at same time a new one is
                       created.
    -s|--source        Override the default source for the trace files with
                       the directory defined with this option
    -m|--mark name     a name (text string)mark such that each line where
                       the string is found will be marked in the trace
                       output with a + as the first character on the line.
                       May be repeated for multiple marks to be placed.
    -s name            trace component which where formatted trace will
                       start. Everything before this is discarded.
                       Default is XmlIO which is logical since it bypasses
                       all of the startup trace and only formats after
                       the first operation is received.
   -x                  Name of trace component where display of output is
                       started. Default: XmlIO
   -c  comment         Comment to be added to summary file
   -t|--tail  count    Keep only the last count lines. Useful when
                       The trace is extremenly large. Combines and sorts
                       before the tail operation but does not reformat
                       until after the tail operation.

This script and its accompaning awk files consolidates and formats a set
of OpenPegasus trace files (one for every process) into a single file sorted
by time and modifies the trace to make it easier to analyze. The modifications
include:
    1. Drop all trace info before the first instance of a specific trace
       component. Default: XmlIO.
    2. Format time relative to this first trace display rather than the
       absolute time
    3. Modify the Process and thread identifiers to more easily identifiable
       names (ex. P01, etc) so that the flow of particular processes and
       threads can be followed.
    4. Mark selected lines of the trace output with a "+" as the first
       character so that they can be easily identified in the trace output.
    5. Format XML output with indents to make it more readable.

The formatting generates two files in the current directory:
   1. Formatted trace file identified with the suffix .fmtd
   2. Summary information file identified with the suffix .txt

All the trace output file names are of the form $OUTFILEPREFIX_<dayOfYear><Time>
EOF
}

#######################################################
## Global variables
#######################################################
DEBUG=0     ## default is 0 meaning off  1 means on
MARKS=""
## name of the Trace component that will start the output.  Anything before this
## will be ignored as the trace is formatted.
StartTraceComponent=""
TraceComment=""
# Parameter for the -t , --tail option. Number of lines at tail to retain
TailSize=""
traceSourceDir=""

#######################################################
##   Functions
#######################################################

## test to confirm that the directory in $1 exists.
## Exit script if it does not exist.
function testPegasusVar() {
    if [ -z "$1" ]; then
        echo $1 " Does not exist"
        exit 1
    fi
}

## read an integer and return as return from the function
## returns value as
## used as follows:
##  READ_FILE_INDEX=$(ReadIntegers maxIntegerSize )
##  echo $READ_FILE_INDEX
## input parameter $1, max size of integer to return.
## this is one based, 0 not allowed.
## The return by echo does not work becasue the echos only return upon
## cmd completion.  For the moment we just set global value.
## alternative is to pass variable name directly

function ReadIntegerValue()
{
    ## test the input $1 and set maxValue appropriately
    local maxValue=$1
    if [[ ! "$maxValue" ]]; then
        exit 1        
    fi

    local __errorFound=1
    local __readResult
    while [ $__errorFound -eq 1 ]
    do
        echo Select 1-$maxValue:
        read __readResult

        ##    echo "Error. Integer input required. Read $__readResult" 
        if  [[ $__readResult != [0-9]* ]]; then
          echo "Error. Integer input required. Read $__readResult"
        else
            echo selected  = $__readResult
            if [ $__readResult -gt $maxValue ]; then
                echo ERROR: Entered Integer $__readResult GT max size of  $maxValue
            elif [ $__readResult -eq 0 ]; then
                echo ERROR: zero not allowed
            else
                __errorFound=0
            fi
        fi
    done

    ## returns by executing echo
    ReadIntegerValueReturn=$__readResult
}

##
## Function that performs the format of a trace file into a formatted
## output file.  The formatted output file has the file extension .fmtd
## $1 is the specification of files to be included in the format process
## Files are located in TraceDir.

function fmtTraceFile()
{
    if [ -z "$traceSourceDir" ]; then
        testPegasusVar PEGASUS_HOME
        TraceDir=$PEGASUS_HOME/trace
    else
        TraceDir=$traceSourceDir
    fi

    if [ -f $TraceDir/cimserver.trc ]; then
        echo Using trace at $TraceDir
    else
        echo No trace cimserver.trc file at $TraceDir
        exit 1
    fi

    if [ -e $OUTFILE.fmtd ]; then
        echo Error the outut file name $OUTFILE.fmtd exists
    else
        echo No file $OUTFILE.fmtd exists. Continue
        rm $OUTFILE.*
        if [ !"$TraceComment" == "" ]; then
            echo "No Comment found"
            echo "No Comment " >FileNames.txt
        else
            echo "Comment " $TraceComment  "found."
            echo $TraceComment >FileNames.txt
        fi
    fi

    ## gather all the input files from the trace directory.
    if [ -n "$1" ]; then
       inputFiles=$1
    else
       echo Get list of files from $TraceDir/"cimserver.trc*"
       inputFiles=$TraceDir/"cimserver.trc*"
    fi

    ## Consolidate the files list into a single date sorted temp file.
    $AWK -f $PROGDIR/consolidateTrace.awk $inputFiles >$OUTFILE_merge.tmp
    echo Single integrated trace file created
    sort -n --key=1.1,1.10n --key=1.12,1.17n $OUTFILE_merge.tmp | tr "\f" "\n" >$OUTFILE.tmp

    echo Single sorted trace file created

    if [ $DEBUG -ne 1 ]; then
        rm $OUTFILE.merge.tmp
    fi

    if [ "$TailSize" != "" ]; then
        ##OrigCount = 
        tail -n $TailSize $OUTFILE.tmp >$OUTFILE1.tmp
        echo Removed all but last $TailSize lines from trace
        rm $OUTFILE.tmp
        mv $OUTFILE1.tmp $OUTFILE.tmp
    fi


    ## set up the variable names to mark. This defines
    ## awk variable keywords with the collection of marks
    markCmdLineVariable=""
    if [ -n "$MARKS" ]; then
        markCmdLineVariable="-v keywords="\"$MARKS\"""
        echo markCmdLineVariable $markCmdLineVariable
    fi
    startTraceCmdLineVar=""
    if [ -n "$StartTraceComponent" ]; then
        startTraceCmdLineVar="-v startElement="\"$StartTraceComponent\"""
        echo setting startTraceCmdLineVar $startTraceCmdLineVar
    fi

    ## Format the consolidated file.
    ## The output is sent to a name defined from the input file name.

    echo $AWK -f $PROGDIR/formatTrace.awk $markCmdLineVariable $startTraceCmdLineVar $OUTFILE.tmp
    $AWK -f $PROGDIR/formatTrace.awk $markCmdLineVariable $startTraceCmdLineVar $OUTFILE.tmp

    if [ $DEBUG -ne 1 ]; then
        rm $OUTFILE.tmp
    fi

    #rename the summary file to match the other names.
    mv FileNames.txt $OUTFILE.SummaryInfo.txt
}

function fixTrace() {
    echo NOT IMPLEMENTED.  Need to complete fixmarks and get trace name
    exit 1
}

#########################################################################
##
##      Main Path of the script
##
#########################################################################

## Test for existence of a command
if [ $# -lt 1 ] ; then
 echo ERROR: At least cmd parameter required
 USAGE
 exit 1
fi

CMD=$1

##
## get all option parameters from the command line
##
if [ $# -gt 1 ] ; then
    shift
    while test -n "$1"; do
        case "$1" in

        --help|-h)
            USAGE
            exit 0
        ;;

        --debug|-d)
            echo debug set
            DEBUG=1
        ;;

        -m|--mark)
            if [ "$2" == "" ]; then
    	        echo value required for parameter $1
    	        exit 1
            fi
            if [ -z $MARKS ]; then
                MARKS=$2
            else
                MARKS=$MARKS,"$2"
            fi
            shift
            echo Marks $MARKS
        ;;

        -x|--xmlComponent)
            if [ $2 == "" ]; then
    	        echo value required for parameter $1
    	        exit 1
            fi

            StartTraceComponent=$2
            shift
            echo Trace component that starts display is $2
        ;;

        -c|--clean)
            echo remove all $OUTFILEPREFIX* files.
            rm $OUTFILEPREFIX*
        ;;

        -s|--source)
            traceSourceDir=$2
            shift
            echo Trace source is $traceSourceDir
        ;;
        
        -C|--comment)
            TraceComment=$2
            shift
            echo Trace Comment = $TraceComment
        ;;
        -t|--tail)
            ## alternate if [ "$2" -eq "$2" ] 2>/dev/null
            if [[ $2 =~ ^-?[0-9]+$ ]]; then
                TailSize=$2
                shift
            else
               echo echo Error: integer required for -t option. $2 found
               exit 1
            fi
            echo Trim trace to include only last $TailSize lines
        ;;
        *)
    	echo "Unknown option $1"
    	USAGE
    	exit 1
    	;;

        esac
        shift
    done
fi

##
## Case statement to execute each of the possible commands
##
case "$CMD" in
    ### list all of the formatted trace files.
    'list') 
        for fn in $OUTFILEPREFIX*.fmtd
        do
           echo $fn
        done
        exit 0
    ;;

    ## remove a specific trace output file.  This function lists all of the
    ## possible files and lets the user select which one to remove
    'remove'|'rm') 
        declare -a names
        for fn in $OUTFILEPREFIX*.fmtd
        do
            ##echo fn $fn
            echo result ${fn%%.*}
            names+=(${fn%%.*})
        done

        ##echo now read for array size ${#names[@]} 
        tLen=${#names[@]}

        for (( i=0; i<${tLen}; i++ ));
        do
            echo $((i+1))  ${names[i]}
        done

        ReadIntegerValue ${#names[@]}     ## change to set per the array size
        index=$(($ReadIntegerValueReturn - 1))
        echo delete $ReadIntegerValueReturn for entry ${names[$index]}
        rm ${names[$index]}*
        exit 0
    ;;

    ## Clean out all existing trace output files.
    'clean')
        echo remove all $OUTFILEPREFIX* files.
        rm $OUTFILEPREFIX*
        exit 0
    ;;

    'help' | '-h' |'--help' | "")
        USAGE
        exit 0
    ;;

##    'fix')
##        fixTrace
##    ;;

    ## format the current trace files
    'format')
        ## Execute the trace format function
        fmtTraceFile
    ;;

    *)
    echo "Unknown command: $CMD"
    USAGE
    exit 1
    	;;
esac