Shell Trying To Output Last Portion Of A Logfile Time Stamp Is The Separator

Find the data you need here

We provide programming data of 20 most popular languages, hope to help you!

Previous PostNext Post

2013-01-10 13:49:55

Notes: 

FAIL: runTest (__main__.changeContent)

Traceback (most recent call last):
  File "demo-about-content-pyunit-w.py", line 294, in runTest
    self.assertEqual(browser.find_element_by_id("descriptionAbout").text, "We are a business used for automated testing and pretty much boring.", 'about desc would not change')
AssertionError: about desc would not change

Ran 1 test in 40.954s>

FAILED (failures=1)
tac logfile.log | while read line; do echo ${line};
    [[ "${line}" =~ [0-9]{4}(-[0-9]{2}){2}\ [0-9]{2}(:[0-9]{2}){2} ]] && break;
done | tac

How to log echo statement with timestamp in shell script

I am currently using following to log stdout and stderr. I want to log echo statements along with timestamp. exec 3>&1 1>>${LOG_FILE} 2>&1 How can I …

exec 3>&1 1>>${LOG_FILE} 2>&1
% echo message | ts
Apr 16 10:56:39 message
% sh
$ alias echo='echo $(date)'
$ echo message
Mon Apr 16 10:57:55 UTC 2018 message
#!/bin/sh

echo() {
    command echo $(date) "[email protected]"
}

echo message
logit() {
    while read
    do
        echo "$(date) $REPLY" >> ${LOG_FILE}
    done
}

LOG_FILE="./logfile"

exec 3>&1 1>> >(logit) 2>&1

echo "Hello world"
ls -l does_not_exist
Mon 16 Apr 2018 09:18:33 BST Hello world
Mon 16 Apr 2018 09:18:33 BST ls: does_not_exist: No such file or directory
# -1 means "current time"
printf "%(%Y-%m-%d %T)T %s\n" -1 "$REPLY"  >> ${LOG_FILE}
2018-04-16 09:26:50 Hello world
2018-04-16 09:26:50 ls: does_not_exist: No such file or directory
logouts() {
    while read
    do
        printf "%(%T)T %s\n" -1 "stdout: $REPLY"  | tee -a ${LOG_FILE}
    done
}
logerrs() {
    while read
    do
        printf "%(%T)T %s\n" -1 "stderr: $REPLY"  >> ${LOG_FILE}
    done
}
LOG_FILE="./logfile"

main()
{
    echo "Hello world" 1>> >(logouts) 2>> >(logerrs)
    ls -l does_not_exist 1>> >(logouts) 2>> >(logerrs)
}
main "[email protected]"

Sort logfile by timestamp on linux command line

The --key=1,2 option tells sort to only use the first two whitespace-separated "fields" (the "freeswitch.log:"-prefixed date, and the time) as the key for sorting. It is important that you specify the last field to use, even if you are only sorting by one field (for instance, if each line kept time and date together in an ISO-8601 standard

...    
freeswitch.log:2011-09-08 12:21:07.282236 [ERR] ftdm_queue.c:136 Failed to enqueue obj 0x7f2cda3525c0 in queue 0x7f2ce8005990, no more room! windex == rindex == 58!
freeswitch.log:2011-08-08 13:21:07.514261 [ERR] ftdm_queue.c:136 Failed to enqueue obj 0x7f2cda354460 in queue 0x7f2ce8005990, no more room! windex == rindex == 58!
freeswitch.log:2011-06-04 16:21:08.998227 [ERR] ftdm_queue.c:136 Failed to enqueue obj 0x7f2cda356300 in queue 0x7f2ce8005990, no more room! windex == rindex == 58! 
freeswitch.log:2011-09-08 12:21:10.374238 [ERR] ftdm_queue.c:136 Failed to enqueue obj 0x7f2cda3581a0 in queue 0x7f2ce8005990, no more room! windex == rindex == 58!
...
sort --stable --reverse --key=1,2 freeswitch.log
freeswitch.log:2011-09-08 12:21:10.374238 Warning: Syntax error on line 20:
freeswitch.log:2011-09-08 12:21:10.374238
freeswitch.log:2011-09-08 12:21:10.374238    My[brackets(call)
freeswitch.log:2011-09-08 12:21:10.374238               ^
freeswitch.log:2011-09-08 12:21:10.374238 Suggestion:
freeswitch.log:2011-09-08 12:21:10.374238   did you forget to
freeswitch.log:2011-09-08 12:21:10.374238   close your brackets?
freeswitch.log:2011-09-08 12:21:10.374238
freeswitch.log:2011-09-08 12:21:10.374238               ^
freeswitch.log:2011-09-08 12:21:10.374238   close your brackets?
freeswitch.log:2011-09-08 12:21:10.374238   did you forget to
freeswitch.log:2011-09-08 12:21:10.374238    My[brackets(call)
freeswitch.log:2011-09-08 12:21:10.374238 Suggestion:
freeswitch.log:2011-09-08 12:21:10.374238 Warning: Syntax error on line 20:
sed 's/:/ /' freeswitch.log | sort -srk2,3 | sed 's/ /:/'
sort -k1 -r freeswitch.log
sort -k1,2 file
while IFS=' ' read -r name_date trailing ; do date=$(cut -d: -f2 <<<"$name_date") ; printf '%s:%s\n' $(date -d "$date" +%s) "$name_date $trailing" ; done < freeswitch.log | sort -k1 -t: | cut -d: -f2-
#!/usr/bin/env bash

logfile="$1"

if [ -f "$logfile" ] ; then
    while IFS=' ' read -r name_date trailing ; do
            date=$(cut -d: -f2 <<<"$name_date")
        printf '%s:%s\n' $(date -d "$date" +%s) "$name_date $trailing"
    done < "$logfile" | sort -k1 -t: | cut -d: -f2-
fi
sort -r
tac yourlogfile

Bash script to log the output of the script to a separate log file

At the same time I want this script to log this output into a log file. Here's my code. echo `date` echo `uptime` And at last . cat /home/rpeb/example.sh 1> /home/rpeb/example.log to redirect the output into a log file. I don't want the last line to be logged. What changes should I make to this code?

echo `date`
echo `uptime`
cat /home/rpeb/example.sh 1> /home/rpeb/example.log 
#!/bin/sh
{ date; uptime; } | tee "$HOME/example.log"
#!/bin/sh
date   | tee    "$HOME/example.log"
uptime | tee -a "$HOME/example.log"
echo `date`
echo `uptime`
echo $(date)
echo $(uptime)
date
uptime
cat /home/rpeb/example.sh 1> /home/rpeb/example.log
cat /home/rpeb/example.sh > /home/rpeb/example.log
chmod u+x /home/rpeb/example.sh
/home/rpeb/example.sh > /home/rpeb/example.log
./example.sh > example.log
./example.sh | tee example.log

How do I search a log file based on timestamp

To make it faster, as your log lines are still sorted by date, you want to search from the restart timestamp to the end of file, so you could set a flag when you find that timestamp and check for the pattern only after that:

dt=$(date +"%D %T")
awk '$0 ~ "Connection refused" && $0 >= $dt' /***.log
[08/06/20 11:36:54.577]:Work...
dt=$(date +"%D %T")
awk -v dt="$dt" '$0 >= dt && $0 ~ /Connection refused/' file
awk -v dt="$dt" 'f && $0 ~ /Connection refused/{print; next} $0 >= dt {f=1}' file
[08/06/20 11:36:54.577]:Work...
awk -v dt="$dt" 'f && $0 ~ /Connection refused/{print; next} substr($0,2) >= dt {f=1}' file

How to extract logs between two time stamps

26. I want to extract all logs between two timestamps. Some lines may not have the timestamp, but I want those lines also. In short, I want every line that falls under two time stamps. My log structure looks like: [2014-04-07 23:59:58] CheckForCallAction [ERROR] Exception caught in +CheckForCallAction :: null --Checking user-- Post [2014-04-08

[2014-04-07 23:59:58] CheckForCallAction [ERROR] Exception caught in +CheckForCallAction :: null
--Checking user--
Post
[2014-04-08 00:00:03] MobileAppRequestFilter [DEBUG] Action requested checkforcall
$ awk -F'[]]|[[]' \
  '$0 ~ /^\[/ && $2 >= "2014-04-07 23:00" { p=1 }
   $0 ~ /^\[/ && $2 >= "2014-04-08 02:00" { p=0 }
                                        p { print $0 }' log
$ awk -F'[]]|[[]' \
  '$0 ~ /^\[/ && $2 >= "2014-04-07 23:00"    { p=1 }
   $0 ~ /^\[/ && $2 >= "2014-04-08 02:00:01" { p=0 }
                                           p { print $0 }' log
$ awk \
  '$0 ~ /^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-2][0-9]:[0-5][0-9]:[0-5][0-9]/
      {
        if ($1" "$2 >= "2014-04-07 23:00")     p=1;
        if ($1" "$2 >= "2014-04-08 02:00:01")  p=0;
      }
    p { print $0 }' log
dategrep --start "12:00" --end "12:15" --format "%b %d %H:%M:%S" syslog
dategrep --end "12:15" --format "%b %d %H:%M:%S" syslog
dategrep --last-minutes 5 --format "%b %d %H:%M:%S" syslog
dategrep --last-minutes 5 --format rsyslog syslog
cat syslog | dategrep --end "12:15" -
[[email protected] ~]$ cat test.txt
Ignore this line, please.
This one too while you're at it...
[2014-04-07 23:59:58] CheckForCallAction [ERROR] Exception caught in +CheckForCallAction :: null
--Checking user--
Post
[2014-04-08 00:00:03] MobileAppRequestFilter [DEBUG] Action requested checkforcall
we don't
want these lines.


[[email protected] ~]$ egrep "^\[2014-04-07 23:59:58\]" test.txt -A 10000 | egrep "^\[2014-04-08 00:00:03\]" -B 10000
[2014-04-07 23:59:58] CheckForCallAction [ERROR] Exception caught in +CheckForCallAction :: null
--Checking user--
Post
[2014-04-08 00:00:03] MobileAppRequestFilter [DEBUG] Action requested checkforcall
CMD=awk -v from=$Date1 -v to=$Date2
error_count=grep "error_message" ${logfile} | awk -F'[ .]' '{print $4}' | $CMD '$1>=from && $1<=to' | wc -l
#!/bin/bash

E_BADARGS=23

if [ $# -ne "3" ]
then
  echo "Usage: `basename $0` \"<start_date>\" \"<end_date>\" file"
  echo "NOTE:Make sure to put dates in between double quotes"
  exit $E_BADARGS
fi 

isDatePresent(){
        #check if given date exists in file.
        local date=$1
        local file=$2
        grep -q "$date" "$file"
        return $?

}

convertToEpoch(){
    #converts to epoch time
    local _date=$1
    local epoch_date=`date --date="$_date" +%s`
    echo $epoch_date
}

convertFromEpoch(){
    #converts to date/time format from epoch
    local epoch_date=$1
    local _date=`date  --date="@$epoch_date" +"%F %T"`
    echo $_date

}

getDates(){
        # collects all dates at beginning of lines in a file, converts them to epoch and returns a sequence of numbers
        local file="$1"
        local state="$2"
        local i=0
        local date_array=( )
        if [[ "$state" -eq "S" ]];then
            datelist=`cat "$file" | sed -r -e "s/^\[([^\[]+)\].*/\1/" | egrep  "^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}"`
        elif [[ "$state" -eq "E" ]];then
            datelist=`tac "$file" | sed -r -e "s/^\[([^\[]+)\].*/\1/" | egrep  "^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}"`

        else
            echo "Something went wrong while getting dates..." 1>&2
            exit 500
        fi

        while read _date
            do
                epoch_date=`convertToEpoch "$_date"`
                date_array[$i]=$epoch_date
                #echo "$_date" "$epoch_date" 1>&2

            (( i++ ))
            done<<<"$datelist"
        echo ${date_array[@]}   


}

findneighbours(){
    # search next best date if date is not in the file using recursivity
    IFS="$old_IFS"
    local elt=$1
    shift
    local state="$1"
    shift
    local -a array=( "[email protected]" ) 

    index_pivot=`expr ${#array[@]} / 2`
    echo "#array="${#array[@]} ";array="${array[@]} ";index_pivot="$index_pivot 1>&2
    if [ "$index_pivot" -eq 1 -a ${#array[@]} -eq 2 ];then

        if [ "$state" == "E" ];then
            echo ${array[0]}
        elif [ "$state" == "S" ];then
            echo ${array[(( ${#array[@]} - 1 ))]} 
        else
            echo "State" $state "undefined" 1>&2
            exit 100
        fi

    else
        echo "elt with index_pivot="$index_pivot":"${array[$index_pivot]} 1>&2
        if [ $elt -lt ${array[$index_pivot]} ];then
            echo "elt is smaller than pivot" 1>&2
            array=( ${array[@]:0:(($index_pivot + 1)) } )
        else
            echo "elt is bigger than pivot" 1>&2
            array=( ${array[@]:$index_pivot:(( ${#array[@]} - 1 ))} ) 
        fi
        findneighbours "$elt" "$state" "${array[@]}"
    fi
}



findFirstDate(){
    local file="$1"
    echo "Looking for first date in file" 1>&2
    while read line
        do 
            echo "$line" | egrep -q "^\[[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}\]" &>/dev/null
            if [ "$?" -eq "0" ]
            then
                #echo "line=" "$line" 1>&2
                firstdate=`echo "$line" | sed -r -e "s/^\[([^\[]+)\].*/\1/"`
                echo "$firstdate"
                break
            else
                echo $? 1>&2
            fi
        done< <( cat "$file" )



}

findLastDate(){
    local file="$1"
    echo "Looking for last date in file" 1>&2
    while read line
        do 
            echo "$line" | egrep -q "^\[[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}\]" &>/dev/null
            if [ "$?" -eq "0" ]
            then
                #echo "line=" "$line" 1>&2
                lastdate=`echo "$line" | sed -r -e "s/^\[([^\[]+)\].*/\1/"`
                echo "$lastdate"
                break
            else
                echo $? 1>&2
            fi
        done< <( tac "$file" )


}

findBestDate(){

        IFS="$old_IFS"
        local initdate="$1"
        local file="$2"
        local state="$3"
        local first_elts="$4"
        local last_elts="$5"
        local date_array=( )
        local initdate_epoch=`convertToEpoch "$initdate"`   

        if [[ $initdate_epoch -lt $first_elt ]];then
            echo `convertFromEpoch "$first_elt"`
        elif [[ $initdate_epoch -gt $last_elt ]];then
            echo `convertFromEpoch "$last_elt"` 

        else
            date_array=( `getDates "$file" "$state"` )
            echo "date_array="${date_array[@]} 1>&2
            #first_elt=${date_array[0]}
            #last_elt=${date_array[(( ${#date_array[@]} - 1 ))]}

            echo `convertFromEpoch $(findneighbours "$initdate_epoch" "$state" "${date_array[@]}")`

        fi

}


main(){
    init_date_start="$1"
    init_date_end="$2"
    filename="$3"
    echo "problem start.." 1>&2
    date_array=( "$init_date_start","$init_date_end"  )
    flag_array=( 0 0 )
    i=0
    #echo "$IFS" | cat -vte
    old_IFS="$IFS"
    #changing separator to avoid whitespace issue in date/time format
    IFS=,
    for _date in ${date_array[@]}
    do
        #IFS="$old_IFS"
        #echo "$IFS" | cat -vte
        if isDatePresent "$_date" "$filename";then
            if [ "$i" -eq 0 ];then 
                echo "Starting date exists" 1>&2
                #echo "date_start=""$_date" 1>&2
                date_start="$_date"
            else
                echo "Ending date exists" 1>&2
                #echo "date_end=""$_date" 1>&2
                date_end="$_date"
            fi

        else
            if [ "$i" -eq 0 ];then 
                echo "start date $_date not found" 1>&2
            else
                echo "end date $_date not found" 1>&2
            fi
            flag_array[$i]=1
        fi
        #IFS=,
        (( i++ ))
    done

    IFS="$old_IFS"
    if [ ${flag_array[0]} -eq 1 -o ${flag_array[1]} -eq 1 ];then

        first_elt=`convertToEpoch "$(findFirstDate "$filename")"`
        last_elt=`convertToEpoch "$(findLastDate "$filename")"`
        border_dates_array=( "$first_elt","$last_elt" )

        #echo "first_elt=" $first_elt "last_elt=" $last_elt 1>&2
        i=0
        IFS=,
        for _date in ${date_array[@]}
        do
            if [ $i -eq 0 -a ${flag_array[$i]} -eq 1 ];then
                date_start=`findBestDate "$_date" "$filename" "S" "${border_dates_array[@]}"`
            elif [ $i -eq 1 -a ${flag_array[$i]} -eq 1 ];then
                date_end=`findBestDate "$_date" "$filename" "E" "${border_dates_array[@]}"`
            fi

            (( i++ ))
        done
    fi


    sed -r -n "/^\[${date_start}\]/,/^\[${date_end}\]/p" "$filename"

}


main "$1" "$2" "$3"

Previous PostNext Post

Related code examples