Calculating Response-Time Statistics with awk

I needed to analyze logs and obtain the minimum, maximum, and average response times for several specified requests.

The Nginx configuration defined this log format:

log_format _main '$remote_addr "$time_iso8601" $request_method "$uri" "$args" "$request_body" $status $body_bytes_sent $request_time "$http_user_agent"';

The second-to-last field, $request_time, is the response time.

Here is an excerpt from the actual log, with the domain replaced:

10.100.27.171 “2015-08-01T00:00:02+08:00” GET “/interface/ipad_v3/sub_channels” “cid=85&ver=3.9.5” “-” 200 588 0.003 “api.3g.example.com/3.0” 10.100.27.172 “2015-08-01T00:00:02+08:00” GET “/interface/ios_v3/sub_channel_details_with_playlist” “sub_channel_id=227&ver=3.9” “-” 200 2912 0.334 “api.3g.example.com/3.0” 10.100.27.172 “2015-08-01T00:00:02+08:00” GET “/interface/android_v3/sub_channel_details” “sub_channel_id=107&show_game_information=1&ver=4.8” “-” 200 3480 0.129 “api.3g.example.com/3.0”

Log filenames followed a pattern similar to log.20150801. I planned to keep the script in the same directory as the logs.

My first version looked like this. Note that $1 can mean both the first shell-function argument and the first field in awk, creating a conflict. The expression '"$1"' closes the single-quoted awk program, inserts shell parameter $1 inside double quotes, and then reopens the single-quoted program. Assigning it with awk -v needle="$1" is another option.

#! /bin/bash

function get_min(){
        ls | grep 'log.' | xargs cat | awk '$7==200 && $4 ~/^"'"$1"'/{print "Min = ", $9}' | sort -n -k 2  |   head -1
}

function get_max() {
        ls | grep 'log.' | xargs cat | awk '$7==200 && $4 ~/^"'"$1"'/{print "Max = ", $9}' | sort -rn -k 2  |   head -1
}

function get_avg() {
        ls | grep 'log.' | xargs cat | awk '$7==200 && $4 ~/^"'"$1"'/{sum+=$9} END {print "Average = ", sum/NR}'
}

This is the most obvious solution, but it is verbose and clearly inefficient. One pass over the files can calculate all three statistics. At the very least, sorting already produces both extremes, so sed can print the first and last records:

function get_min_and max {
    ls | grep 'log.' | xargs cat | awk '$8=200 && $4 ~/^"'"$1"'/{print $9}' | sort -n -k 2  |   sed -n '1p;$p'
}

After studying awk more carefully, I arrived at this version:

#! /bin/bash

function cal_work(){
        ls | grep 'log.' | xargs cat | awk   'BEGIN{max=0;min=1}
                {if ($4 ~ /^"'"$1"'/  && $(NF-3)==200){
                        sum+=$9; count+=1;
                        if($9 > max) max=$9 fi;if($9 < min) min=$9 fi;
                    }
                }
                END {print "Average = ", sum/count;print "Max = ", max;print "Min", min}'
}

function get_stat(){
        target=$1
        target=${target//\//\\/}
        cal_work $target
}

get_stat '/interface/ipad_v3/sub_channels'

The path being queried contains slashes, so each slash must be escaped with \. get_stat performs that conversion.

Originally published on SegmentFault.