Aggregating Container Logs | Administration

Overview
Using a Centralized File System
- Installing fluentd (td-agent) on Nodes
- Optional Method to Verify Working Nodes

Overview

As an OpenShift administrator, you may want to view the logs from all containers in one user interface. The currently supported method for aggregating container logs in OpenShift Enterprise is using a centralized file system. Additional supported methods are planned for inclusion in future releases.

As packaging improvements are made, these instructions will be simplified.

Using a Centralized File System

This method reads all container logs and forwards them to a central server for storage on the file system.

Installing fluentd (td-agent) on Nodes

Perform the following steps on each node to install and configure fluentd (td-agent):

Run the following commands:

# export RPM=td-agent-2.2.0-0.x86_64.rpm
# curl https://packages.treasuredata.com/2/redhat/7/x86_64/$RPM \
    -o /tmp/$RPM
# yum localinstall /tmp/$RPM
# /opt/td-agent/embedded/bin/gem install fluent-plugin-kubernetes
# mkdir -p /etc/td-agent/config.d
# chown td-agent:td-agent /etc/td-agent/config.d

Create a directory to house the logs:

# mkdir -p /var/log/td-agent/tmp
# chown td-agent:td-agent /var/log/td-agent/tmp

To allow td-agent access to the containers logs, create the /etc/sysconfig/td-agent file and ensure it contains the following:

DAEMON_ARGS=
TD_AGENT_ARGS="/usr/sbin/td-agent --log /var/log/td-agent/td-agent.log --use-v1-config"

Add the following line to the /etc/td-agent/td-agent.conf file:
@include config.d/*.conf

Create the /etc/td-agent/config.d/kubernetes.conf file with the following contents:

    <source>
      type tail
      path /var/lib/docker/containers/*/*-json.log
      pos_file /var/log/td-agent/tmp/fluentd-docker.pos
      time_format %Y-%m-%dT%H:%M:%S
      tag docker.*
      format json
      read_from_head true
    </source>

    <match docker.var.lib.docker.containers.*.*.log>
      type kubernetes
      container_id ${tag_parts[5]}
      tag docker.${name}
    </match>

    <match kubernetes>
      type copy
      <store>
        type forward
        send_timeout 60s
        recover_wait 10s
        heartbeat_interval 1s
        phi_threshold 16
        hard_timeout 60s
        log_level trace
        require_ack_response true
        heartbeat_type tcp
        <server>
          name logging_name (1)
          host host_name (2)
          port 24224
          weight 60
        </server>

        <secondary>
          type file
          path /var/log/td-agent/forward-failed
        </secondary>
      </store>

      <store>
        type file
        path /var/log/td-agent/containers.log
        time_slice_format %Y%m%d
        time_slice_wait 10m
        time_format %Y%m%dT%H%M%S%z
        compress gzip
        utc
      </store>
    </match>

1	The name for the master that will be used during logging.
2	The IP or a DNS resolvable name used to access the master.

Enable fluentd:

# systemctl enable td-agent
# systemctl start td-agent

Any errors are logged in the /var/log/td-agent/td-agent.log file.

Optional Method to Verify Working Nodes

You can optionally set up the master to be the aggregator to test and verify that the nodes are working properly.

Install fluentd (td-agent) on the master:

# export RPM=td-agent-2.2.0-0.x86_64.rpm
# curl https://packages.treasuredata.com/2/redhat/7/x86_64/$RPM \
    -o /tmp/$RPM
# yum localinstall /tmp/$RPM
# mkdir -p /etc/td-agent/config.d
# chown td-agent:td-agent /etc/td-agent/config.d

Ensure port 24224 is open on the master’s firewall to allow the nodes access.
Configure fluentd to aggregate container logs by adding the following line to the /etc/td-agent/td-agent.conf file:
@include config.d/*.conf

Create the /etc/td-agent/config.d/kubernetes.conf file with the following contents:

    <match kubernetes.**>
        type file
        path /var/log/td-agent/containers.log
        time_slice_format %Y%m%d
        time_slice_wait 10m
        time_format %Y%m%dT%H%M%S%z
        compress gzip
        utc
    </match>

Enable fluentd:
# systemctl enable td-agent # systemctl start td-agent
Any errors are logged in the /var/log/td-agent/td-agent.log file.

You should now find all the containers' logs available on the master in the /var/log/td-agent/containers.log file.