Classes

Abstract Class: fake_log_gen

For the fake log generator, according to the Log Patterns, currently we want to generate 2 different formats of logs: Apache access log & Apache error log. To separate them, Visor created an abstract class fake_log_gen, then inherit from this to separately create fake_access_gen and fake_error_gen.

Subclass: fake_access_gen

fake_access_gen is used to generate logs in Apache access format. First, inherit from the fake_log_gen:

class fake_access_gen(fake_log_gen):

During its initialization, assign the loaded configuration information to the class variables:

def __init__(self, log, config, mode):
        self.log = log
        self.mode = mode
        # Dict that contains config info
        self.config = config

        self.access_min = self.config["access"]["interval"]["min"]
        self.access_max = self.config["access"]["interval"]["max"]
        self.user_ids = self.config["access"]["user_id"]
        self.methods = self.config["access"]["method"]
        self.resources = self.config["access"]["resource"]
        self.codes = self.config["access"]["code"]
        self.versions = self.config["access"]["version"]

Heartbeat Line

For the Apache access log, currently we set it to have 2 kinds of logs: the heartbeat and the normal access logs. Heartbeat is simple, just emit it every once in a while at a set interval.

- - - [28/Dec/2016:16:55:53 -0700] "HEARBEAT" - -

Since it's meaningless for heartbeat logs to have user ids and other fields, use - to fill in these fields. The time field is formatted as %d/%b/%Y:%H:%M:%S, so the entire code will be:

@coroutine
    def heartbeat_lines(self):
        while True:
            t = datetime.datetime.now().strftime('%d/%b/%Y:%H:%M:%S -0700')
            self.log.info('- - - [%s] "%s" - -', t, self.config["heartbeat"]["message"])
            yield from asyncio.sleep(int(self.config["heartbeat"]["interval"]))

The logging and asycio part will be discussed later in detail.

Access Line

The access lines contain most of the information we want. Unlike heartbeat lines, we don't know when the next access line will come, so we set a range of intervals to randomize their coming times:

yield from asyncio.sleep(random.uniform(self.access_min, self.access_max))

More details and more reasonable randomization method will be discussed later.

Access lines contains fields like: ip address, user identity, user id, time, request line message, http status code and the size of the object returned to the client. For most of them, we can randomly pick the pre-set contents from the configuration information for a single generated log line, for example:

code = self.codes[random.randint(0, len(self.codes)-1)]

Subclass: fake_error_gen

fake_error_gen is used to generate logs in Apache error format. Similar to access log, inherit from the fake_log_gen:

class fake_error_gen(fake_log_gen):

The __init__ method is also very similar to access log's, just assign the required configuration information to class variables.

Heartbeat Line

Same concept with access log, leave fields to - except time and information level:

[Wed Dec 28 18:55:35 2016] [INFO] [-] [-] HEARBEAT

Warn Line

The error logs have several different message levels, to indicate the relative importance of them. The convention is that INFO means the most general information, DEBUG means diagnostically helpful information, WARN means anything that can potentially cause problems, ERROR just refer errors, and some other levels.

Here in Visor, we currently use 3 of these levels: INFO, WARN and ERROR. All heartbeat lines belong to INFO.

For the WARN lines, since we just want to simulate a cluster that generate logs, we can just use some interesting messages:

[Wed Dec 28 18:55:35 2016] [WARNING] [pid 26712:tid 1300805056] [client 13.195.146.243] My phone is out of battery

Error Line

The ERROR lines are generated just the same way as WARN lines, except that the contents of fields would be different.

[Wed Dec 28 18:55:35 2016] [ERROR] [pid 37787:tid 0865979496] [client 151.238.217.124] Premature end of script headers