Extensible Log Management

This document describes how to extend the log management capabilities of thin-edge.io using custom log plugins. By default, thin-edge.io supports retrieving file-based logs. But, it can be extended with plugins to collect logs from other sources like systemd journal, docker containers or any custom log sources.

Overview

The thin-edge.io agent supports extensible log management through a plugin system:

• The built-in file plugin handles traditional file-based logs (the default behavior)
• Additional plugins can be installed to support other log sources
• Each plugin can provide multiple log types
• Plugins are discovered and executed automatically by tedge-agent

How Log Plugins Work

A log plugin is an executable that implements a simple command-line interface with the following requirements:

• Implement two sub-commands:
  1. list - Returns all log types supported by the plugin (one per line)
  2. get <log-type> [--since <timestamp>] [--until <timestamp>] - Retrieves logs for the specified type within the given time range
• Must exit with code 0 for successful list command (used to validate the plugin).
• Should output logs to stdout for the get command.
• Time filters --since and --until are passed as seconds since epoch.
• Should handle errors gracefully and exit with non-zero codes on failure.

The agent automatically:

• Discovers plugins at startup by running their list command to gather their supported log types.
• Publishes the supported log types to the device's log_upload command metadata topic (e.g: te/device/main///cmd/log_upload) with a plugin suffix in the format <log-type>::<plugin-name> (e.g., mosquitto::journald)
• Routes log_upload requests to the get command of the appropriate plugin based on the type suffix.
• The dateFrom and dateTo parameters in the command are passed to the plugin as --since and --until arguments.
• Further filtering by searchText and tail lines are done by the agent itself.
• Detects any new plugin installations dynamically.
• Refresh the supported log types by reloading the plugins when any new software is installed or configuration is updated.

Log plugins are installed at /usr/share/tedge/log-plugins, by default.

This plugin root directory can be changed using:

sudo tedge config set log.plugin_paths /usr/local/share/tedge/log-plugins,/usr/share/tedge/log-plugins

Multiple plugin directories can be specified if a layered directory structure is desired, where plugins from directories earlier in the list gets precedence over the latter ones. For example, with the above configuration, a file plugin in /usr/local/share/tedge/log-plugins would override the one with the same name in /usr/share/tedge/log-plugins. This mechanism can be used to override the factory plugins when needed.

Permissions

Plugins are executed by the agent with sudo privileges. The agent automatically creates the following sudoers entry, giving sudo rights to all plugins installed at /usr/local/lib/tedge/log-plugins:

tedge    ALL = (ALL) NOPASSWD:SETENV: /usr/local/lib/tedge/log-plugins/[a-zA-Z0-9]*

If the log.plugin_paths config is updated with additional directories as shown in the previous section, then sudoers entries must be created for those directories as well.

Additionally, ensure your plugin has appropriate permissions to access the log sources it needs.

Creating a Custom Log Plugin

Example: docker plugin

Here's a docker plugin example that can retrieve logs from containers using the docker logs command:

#!/bin/sh
set -eu

help() {
    cat <<EOT
docker log plugin to retrieve the logs from containers using the docker cli

$0 <SUBCOMMAND>

SUBCOMMANDS
  list
  get <type> [--since <timestamp>] [--until <timestamp>]
EOT
}

list_log_types() {
    docker ps -a --format "{{.Names}}"
}

get_log_by_type() {
    log_type="$1"
    shift

    # Parse option defaults
    since="24h"
    until="0s"

    while [ $# -gt 0 ]; do
        case "$1" in
            --since)
                since="$2"
                shift
                ;;
            --until)
                until="$2"
                shift
                ;;
        esac
        shift
    done

    # Retrieve logs using docker logs
    docker logs "$log_type" \
        --since "$since" \
        --until "$until"
}

if [ $# -lt 1 ]; then
    echo "Missing required subcommand" >&2
    help
    exit 1
fi

SUBCOMMAND="$1"
shift

case "$SUBCOMMAND" in
    list)
        list_log_types
        ;;
    get)
        get_log_by_type "$@"
        ;;
    *)
        echo "Unsupported command" >&2
        exit 1
        ;;
esac

• The list command of this plugin will output all container names:

  nginx
  mosquitto

• The get command retrieves the logs for the target container using the docker logs command.

Installation

Copy the plugin to the plugins directory and make it executable:

sudo cp /path/to/docker /usr/share/tedge/log-plugins/
sudo chmod +x /usr/share/tedge/log-plugins/docker

Testing the Plugin

List all log types the plugin supports:

sudo /usr/share/tedge/log-plugins/docker list

Retrieve logs for a specific service:

sudo /usr/share/tedge/log-plugins/docker get ssh

Retrieve logs with time filters (timestamps in seconds since epoch):

sudo /usr/share/tedge/log-plugins/docker get tedge-agent --since 1696250000 --until 1696260000

Dynamic log type discovery

To ensure that any newly installed services or log sources are immediately available for log collection, tedge-agent automatically reloads the plugins and refreshes the supported log types on the following events:

• A new log plugin is installed in the plugin directory (/usr/share/tedge/log-plugins)
• The tedge-log-plugin.toml file is updated
• A new software is installed with the agent (via the software_update command)
• A new configuration is installed or updated with the agent (via the config_update command)

If the new software or configuration is installed/updated externally, and not via tedge-agent, a refresh can be triggered by just touching the plugin configuration file:

touch /etc/tedge/plugins/tedge-log-plugin.toml

Factory Plugins

• The default file plugin is included in the tedge installation package itself on all distributions.
• A journald plugin that can gather systemd service logs using the journalctl command is also included in the tedge packages for systemd based distributions like Debian, Ubuntu, RHEL etc.