log

Logging functions and classes

Note

Best practices retrieved from Python Logging Cookbook

append

LogAppend

A custom class (sub-class of LogAdapter) is also included in the log module to provide an option to automatically append data to every log message (e.g. a Session Id, Username of the executor).

This is normally done through a logging.LogAdapter(), however, it uses the extra keyword argument and if the LogAdapter and the individual logging call with extra keyword argument is provided, the results are not combined. This custom class allows for merging of the two.

To use the custom class, add the following (usually after initial setting of logging settings at initialization):

from elevaso_spine.log import append

user = "me"

LOGGER = append.LogAppend(LOGGER, {'user': user, 'session_id': 'testing session id'})

Now, every logging message will contain a user and session_id key with the values provided to the LogAppend class

class elevaso_spine.log.append.LogAppend(logger: Logger, extra: dict, **kwargs)[source]

Custom class for managing appended data to all log messages

__init__(logger: Logger, extra: dict, **kwargs)[source]

Initialize an instance of the LogAppend class

Args:

logger (logging.Logger): Logger to associate with this class

extra (dict): dictionary of values to append to each log record

Kwargs:

adapter_priority: If key exists in log message and adapter, does the adapter have priority, defaults to False

process(msg: str, kwargs: dict) → Tuple[str, dict][source]

Process the CustomAdapter log message and append extra values if provided

Note

This actually processes the logging message, the custom handler is needed because if you pass in extra into the LoggerAdapter and extra into the individual logging, the standard Python LoggerAdapter will ignore/overwrite the logging extra.

Args:

msg (str): Message for the log

kwargs (dict): Dictionary of keyword arguments

Note

Normally we would use **kwargs, however, the logging process function passes in as a positional argument

Returns:

msg (str): Message for the log

kwargs (dict): Updated keyword arguments

config

setup

The elevaso_spine.log.config.setup() provides a quick way to configure logging through Python code or a JSON file.

Default Config

elevaso_spine includes default log configuration files located in elevaso_spine/log/data and can easily be referenced through their short name in Python code.

Below is an example of how to use the default configuration for JSON logging.

from elevaso_spine.log import config

config.setup(log_format="json")

Optionally, you can override the default log configuration level by providing the log_level argument.

from elevaso_spine.log import config

config.setup(log_format="json", log_level="WARN")

Custom Config

To use your own log config file in your project, create a .json file that resembles the following:

{
    "version": 1,
    "disable_existing_loggers": false,
    "formatters": {
        "json":{
            "include_keys":["message", "module", "levelname", "name", "funcName", "asctime", "thread", "threadName"],
            "()":"elevaso_spine.log.fmt_json.JsonFormatter",
            "timestamp_key": "timestamp",
            "session_key":"init_id",
            "timezone":"US/Central",
            "dtm_format": "%Y-%m-%d %H:%M:%S %Z"
        }
    },
    "handlers": {
        "console": {
            "class": "logging.StreamHandler",
            "level": "DEBUG",
            "formatter": "json",
            "stream": "ext://sys.stdout"
        }
    },
        "root": {
        "level": "INFO",
        "handlers": ["console"],
        "propogate":"no"
    }
}

In your python code, when calling the elevaso_spine.log.config.setup() function, provide the path to the .json file in your project.

import os
from elevaso_spine.log import config

config.setup(path=os.path.join(os.path.dirname(__file__), "log.json"))

Python Config - Functions

If you would rather setup logging without a configuration file, you can set this up directly in your Python code.

import logging
from elevaso_spine.log import append, fmt_json

LOGGER = logging.getLogger(__name__)

LOGGER.setLevel(logging.DEBUG)
log_handler = logging.StreamHandler()
LOGGER.addHandler(log_handler)

# Create a JSON Formatter instance
formatter = fmt_json.JsonFormatter()

# Add JSON Formatter to Log Handler
log_handler.setFormatter(formatter)

Optional arguments can be provided to the Formatter class. See elevaso_spine.log.fmt.BaseFormatter() for options.

Python Config - Dictionary

Another method of setting the configuration directly in Python is to use a dict.

 import logging
 import logging.config

 log_config = {
     "version": 1,
     "disable_existing_loggers": false,
     "formatters": {
         "json":{
             "include_keys":["message", "module", "levelname", "name", "funcName", "asctime", "thread", "threadName"],
             "()":"elevaso_spine.log.fmt_json.JsonFormatter",
             "timestamp_key": "timestamp",
             "session_key":"init_id",
             "timezone":"US/Central",
             "dtm_format": "%Y-%m-%d %H:%M:%S %Z"
         }
     },
     "handlers": {
         "console": {
             "class": "logging.StreamHandler",
             "level": "DEBUG",
             "formatter": "json",
             "stream": "ext://sys.stdout"
         }
     },
         "root": {
         "level": "INFO",
         "handlers": ["console"],
         "propogate":"no"
     }
 }

 LOGGER = logging.getLogger(__name__)

 logging.config.dictConfig(log_config)

elevaso_spine.log.config.setup(path: str = None, log_format: str = None, log_config: dict = None, **kwargs)[source]

Function to setup initial logging configuration

Args:

path (str, Optional): Path to the directory of the logging config, defaults to calling object path & data directory

Note

If log_config is None and path is None, then the path to the calling object will be used as the root directory. This function will look for a file named log_config (either .json, .yml, .yaml) in /data/directory

log_format (str, Optional): Format of the logging (standard or json), defaults to None

Note

If log_format is None, it will attempt to load from the environment variable LOG_FORMAT. If that does not exist, standard will be used as the default value

log_config (dict, Optional): Logging configuration (to override defaults and not retrieve from a file), defaults to None

Kwargs:

log_level (str, Optional): Log level to set the global LOGGER to, defaults to None

Note

If log_level is None, it will attempt to load from the environment variable LOG_LEVEL. If that does not exist, INFO will be used as the default value

fmt

BaseFormatter

The elevaso_spine.log.fmt.BaseFormatter() class provides the foundational functions for other log formatters. It includes:

Date/time formatting
Timezone support (defaults to UTC)
Support for the extra keyword argument

LOGGER.info('testing', extra={'test1': 'hello', 'test2': 'there'})

Session identifier (defaults to UUID if none provided)

class elevaso_spine.log.fmt.BaseFormatter(**kwargs)[source]

Custom base formatting class

Attributes:

default_dtm_format: Default date/time format, defaults to %Y-%m-%d %H:%M:%S.%f %z

default_time_zone: The default time zone to output timestamps in, defaults to UTC

__init__(**kwargs)[source]

Initialize instance of the class

Kwargs:

include_keys (list): List of reserved keys to include in all json logging, defaults to RESERVED_ATTRS

timestamp_key (str): Name of the timestamp json key (either provided in log message extra or automatically added), defaults to None

session_key (str): Name of the session json key (either provided in log message extra or automatically added as uuid4), defaults to None

dtm_format (str): Date/time format for asctime and/or timestamp_key, defaults to self.default_dtm_format

timezone (str): String representing the time zone, defaults to self.default_timezone

extra (dict): Dictionary of key/value data to provide on all log messages

Raises:

ValueError if timezone is invalid

format(record: LogRecord) → str[source]

Formats the log record

Args:: record (logging.LogRecord): log record to format
Returns:: object: Formatted log output

formatTime(record: LogRecord, datefmt: str = None) → str[source]

Format the time and converts to desired timezone

Args:

record (logging.LogRecord): log record to format

datefmt (str, optional): Format of the date/time, defaults to None

Returns:

str: String of the date/time value

fmt_json

JsonFormatter

The elevaso_spine.log.fmt_json.JsonFormatter() inherits from the elevaso_spine.log.fmt.BaseFormatter() but provides an output function to convert logs into JSON format.

Using the default JSON logging configuration file and setup from Default Config and running the following code

LOGGER.info('test')

will output something like:

{"name": "root", "msg": "test", "args": [], "levelname": "INFO", "levelno": 20, , "filename": "log.py", "module": "log", "exc_info": null, "exc_text": null, "stack_info": null, "lineno": 211, "funcName": "<module>", "created": 1575907675.176455, "msecs": 176.45502090454102, "relativeCreated": 1.3270378112792969, "thread": 4610047424, "threadName": "MainThread", "processName": "MainProcess", "process": 62034, "message": "test", "asctime": "2019-12-09 16:07:55.176455 +0000"}

To add additional key/value pairs, you can use the extra keyword argument in the logger:

LOGGER.info('testing', extra={'test1': 'hello', 'test2': 'there'})

or provide the raw JSON/dictionary object type to the logger:

LOGGER.info({'test1': 'hello', 'test2': 'there'})

Note

The only difference between the two methods is that the first contains a msg key (with the value of testing) whereas the second does not (since the message was only the contents of the extra keyword arguments).

The other difference is that if you decided to not format the output in Json format (e.g. using a standard CLI print type format), the extra keyword arguments are not automatically printed to the console, so only the values in the message (in the first usage example) will be shown. For example:

import logging

logging.basicConfig(
    level=logging.DEBUG,
    format="...%(asctime)s...%(message)s",
    datefmt="%Y-%m-%d %H:%M:%S",
)

LOGGER.info({'test1': 'hello', 'test2': 'there'})
LOGGER.info('testing', extra={'test1': 'hello', 'test2': 'there'})

will output:

...2019-12-09 10:21:50...{'test1': 'hello', 'test2': 'there'}
...2019-12-09 10:21:50...testing

This problem can be overcome by using the formatter_standard.StandardFormatter class, which is intended for terminal output, but will include an extra keyword arguments in a standard format

class elevaso_spine.log.fmt_json.JsonFormatter(**kwargs)[source]

Custom class for logging in Json format

format_func(val: dict, **kwargs) → str[source]

Final output formatting function

Args:: val (dict): Dictionary of values to output in JSON
Returns:: str: JSON string

output_func(val: dict, **kwargs) → str

Final output formatting function

Args:: val (dict): Dictionary of values to output in JSON
Returns:: str: JSON string

fmt_standard

StandardFormatter

The elevaso_spine.log.fmt_standard.StandardFormatter() inherits from the elevaso_spine.log.fmt.BaseFormatter() but provides an output function to convert logs in standard (i.e. Command Line Interface) format.

The benefit to using this formatter is that the extra keyword arguments are displayed, allowing you to easily switch between JSON and standard (CLI) formatting.

A logging message like:

LOGGER.info('testing', extra={'test1': 'hello', 'test2': 'there'})

would be displayed as:

[INFO]2024-04-19 10:21:50..module.function..testing || Extra || [test1 || hello]..[test2 || there]

class elevaso_spine.log.fmt_standard.StandardFormatter(**kwargs)[source]

Custom class for logging in a standard format

format_func(val: dict, **kwargs) → str[source]

Final output formatting function

Args:: val (dict): Dictionary of values to output
Kwargs:: extra_keys (list): List of keys that were included in the extra
Returns:: str: String to print

output_func(val: dict, **kwargs) → str

Final output formatting function

Args:: val (dict): Dictionary of values to output
Kwargs:: extra_keys (list): List of keys that were included in the extra
Returns:: str: String to print