Code Details

This chapter will discuss about the code details of the mini-monitor.

Arguments

-i argument is the input log file for monitoring.

python3.4 $Visor_HOME/src/mini_monitor/mini_monitor.py -i fake_error_file.log

For the concepts, check fake- generator's Arguments chapter, the implementation is very similar.

Class

We create a mini_monitor class to achieve most of the monitor's functions.

On initialization, the mini_monitor object will accept the input log file name, and then open it for analysis.

# Instantiate the monitor
monitor = mini_monitor(logfile)
# Start the monitoring
monitor.analyze()

def __init__(self, logfile):
    self.logfile = logfile

Read Log Lines

The mini_monitor will read the log files line by line, also, the fake_log_generator will write into the log file in append mode. Thus, the system will support that mini_monitor keeps monitoring any changes to the log file written by the fake_log_generator.

This kind of continuous monitoring is implemented in analyze() method:

def analyze(self):
        with open(self.logfile, 'r') as f:
            while True:
                # Remove the trailing \n
                line = f.readline().rstrip()
                if line == '' or line == '\n':
                    time.sleep(0.1)
                else:
                    self.on_data(line)

If the monitor reaches the end of the file, it will check every 0.01 sec to see if there are any new lines added.

Log Format Checking

When the mini-monitor imports the log lines, it doesn't know which log format they are, so some checking work needs to be done before analysis.

Currently, there are 2 possible formats:

• Access log format:

  232.48.164.85 - Frank [03/Jan/2017:11:24:57 -0700] "DELETE /doc/charge.txt HTTP/2.0" 200 1740
  

• Error log format:

  [Tue Jan 03 11:39:58 2017] [WARNING] [pid 40464:tid 4449765814] [client 123.58.12.172] My phone is out of battery
  

So, just check the first character of the log line can be enough:

if line[0] == '[':
    self.on_error_log(line)
else:
    self.on_access_log(line)

Error Reporting

In this prototype, we first implement a small feature: error reporting.

From the previous paragraph, when a log line is determined to be an error format line, the class method on_error_log() will be called and that log line will also be passed to the method.

In this method, further analysis should be performed to see if this is a formal INFO line, a WARNING line, or an ERROR line that we hope to be reported. So we want to extract the log level field from each error format line.

Take a look at the error log format, every field except the message detail is enclosed by []. So we can split the log line string by [ or ]:

def on_error_log(self, line):
    fields = line.split(']')
    fields = [x.strip(' [') for x in fields]
    ...

Now fields[1] will contain the log level message. If it's an ERROR level line, invoke another method to report the error:

if lv == 'ERROR':
    self.error_report(fileds)

Report Format

At this step for this mini-monitor, just want to show the cumulative number of errors occurred so far, and the detail message of this error line.

So, define a class variable to store the total error count in the __init__ method:

self.error_cnt = 0

Then in the invoked error_report() method:

def error_report(self, fields):
    self.error_cnt += 1
    print('Total num of errors: ' + str(self.error_cnt))
    print('Error detail: ' + fields[-1])

Console output example:

Total num of errors: 1
Error detail: Premature end of script headers