Logging References
Collector features
| Output | Protocol | Tested with | Fluentd | Vector |
|---|---|---|---|---|
Cloudwatch |
ReST over HTTP(S) |
✓ |
✓ |
|
elasticsearch v6 |
v6.8.1 |
✓ |
✓ |
|
elasticsearch v7 |
v7.12.2, 7.17.7 |
✓ |
✓ |
|
elasticsearch v8 |
v8.4.3 |
✓ |
||
Fluent Forward |
Fluentd forward v1 |
Fluentd 1.14.6, Logstash 7.10.1 |
✓ |
|
Google Cloud Logging |
✓ |
|||
HTTP |
HTTP 1.1 |
Fluentd 1.14.6, Vector 0.21 |
||
Kafka |
Kafka 0.11 |
Kafka 2.4.1, 2.7.0, 3.3.1 |
✓ |
✓ |
Loki |
ReST over HTTP(S) |
Loki 2.3.0, 2.7 |
✓ |
✓ |
Splunk |
HeC |
v8.2.9, 9.0.0 |
✓ |
|
Syslog |
RFC3164, RFC5424 |
Rsyslog 8.37.0-9.el7 |
✓ |
| Feature | Fluentd | Vector |
|---|---|---|
App container logs |
✓ |
✓ |
App-specific routing |
✓ |
✓ |
App-specific routing by namespace |
✓ |
✓ |
Infra container logs |
✓ |
✓ |
Infra journal logs |
✓ |
✓ |
Kube API audit logs |
✓ |
✓ |
OpenShift API audit logs |
✓ |
✓ |
Open Virtual Network (OVN) audit logs |
✓ |
✓ |
| Feature | Fluentd | Vector |
|---|---|---|
elasticsearch certificates |
✓ |
✓ |
elasticsearch username / password |
✓ |
✓ |
Cloudwatch keys |
✓ |
✓ |
Cloudwatch STS |
✓ |
✓ |
Kafka certificates |
✓ |
✓ |
Kafka username / password |
✓ |
✓ |
Kafka SASL |
✓ |
✓ |
Loki bearer token |
✓ |
✓ |
| Feature | Fluentd | Vector |
|---|---|---|
Viaq data model - app |
✓ |
✓ |
Viaq data model - infra |
✓ |
✓ |
Viaq data model - infra(journal) |
✓ |
✓ |
Viaq data model - Linux audit |
✓ |
✓ |
Viaq data model - kube-apiserver audit |
✓ |
✓ |
Viaq data model - OpenShift API audit |
✓ |
✓ |
Viaq data model - OVN |
✓ |
✓ |
Loglevel Normalization |
✓ |
✓ |
JSON parsing |
✓ |
✓ |
Structured Index |
✓ |
✓ |
Multiline error detection |
✓ |
|
Multicontainer / split indices |
✓ |
✓ |
Flatten labels |
✓ |
✓ |
CLF static labels |
✓ |
✓ |
| Feature | Fluentd | Vector |
|---|---|---|
Fluentd readlinelimit |
✓ |
|
Fluentd buffer |
✓ |
|
- chunklimitsize |
✓ |
|
- totallimitsize |
✓ |
|
- overflowaction |
✓ |
|
- flushthreadcount |
✓ |
|
- flushmode |
✓ |
|
- flushinterval |
✓ |
|
- retrywait |
✓ |
|
- retrytype |
✓ |
|
- retrymaxinterval |
✓ |
|
- retrytimeout |
✓ |
| Feature | Fluentd | Vector |
|---|---|---|
Metrics |
✓ |
✓ |
Dashboard |
✓ |
✓ |
Alerts |
✓ |
| Feature | Fluentd | Vector |
|---|---|---|
Global proxy support |
✓ |
✓ |
x86 support |
✓ |
✓ |
ARM support |
✓ |
✓ |
IBM Power support |
✓ |
✓ |
IBM Z support |
✓ |
✓ |
IPv6 support |
✓ |
✓ |
Log event buffering |
✓ |
|
Disconnected Cluster |
✓ |
✓ |
Logging 5.6 API reference
ClusterLogForwarder
ClusterLogForwarder is an API to configure forwarding logs.
You configure forwarding by specifying a list of pipelines,
which forward from a set of named inputs to a set of named outputs.
There are built-in input names for common log categories, and you can define custom inputs to do additional filtering.
There is a built-in output name for the default openshift log store, but you can define your own outputs with a URL and other connection information to forward logs to other stores or processors, inside or outside the cluster.
For more details see the documentation on the API fields.
| Property | Type | Description |
|---|---|---|
spec |
object |
Specification of the desired behavior of ClusterLogForwarder |
status |
object |
Status of the ClusterLogForwarder |
.spec
Description
ClusterLogForwarderSpec defines how logs should be forwarded to remote targets.
Type
-
object
| Property | Type | Description |
|---|---|---|
inputs |
array |
(optional) Inputs are named filters for log messages to be forwarded. |
outputDefaults |
object |
(optional) DePReCATeD OutputDefaults specify forwarder config explicitly for the default store. |
outputs |
array |
(optional) Outputs are named destinations for log messages. |
pipelines |
array |
Pipelines forward the messages selected by a set of inputs to a set of outputs. |
.spec.inputs[].application
.spec.outputDefaults.elasticsearch
Description
elasticsearchStructuredSpec is spec related to structured log changes to determine the elasticsearch index
Type
-
object
| Property | Type | Description |
|---|---|---|
enableStructuredContainerLogs |
bool |
(optional) enableStructuredContainerLogs enables multi-container structured logs to allow |
structuredTypeKey |
string |
(optional) StructuredTypeKey specifies the metadata key to be used as name of elasticsearch index |
structuredTypeName |
string |
(optional) StructuredTypeName specifies the name of elasticsearch schema |
.spec.outputs[]
Description
Output defines a destination for log messages.
Type
-
array
| Property | Type | Description |
|---|---|---|
syslog |
object |
(optional) |
fluentdForward |
object |
(optional) |
elasticsearch |
object |
(optional) |
kafka |
object |
(optional) |
cloudwatch |
object |
(optional) |
loki |
object |
(optional) |
googleCloudLogging |
object |
(optional) |
splunk |
object |
(optional) |
name |
string |
Name used to refer to the output from a |
secret |
object |
(optional) Secret for authentication. |
tls |
object |
TLS contains settings for controlling options on TLS client connections. |
type |
string |
Type of output plugin. |
url |
string |
(optional) URL to send log records to. |
.spec.pipelines[]
Description
PipelinesSpec link a set of inputs to a set of outputs.
Type
-
array
| Property | Type | Description |
|---|---|---|
detectMultilineerrors |
bool |
(optional) DetectMultilineerrors enables multiline error detection of container logs |
inputRefs |
array |
InputRefs lists the names ( |
labels |
object |
(optional) Labels applied to log records passing through this pipeline. |
name |
string |
(optional) Name is optional, but must be unique in the |
outputRefs |
array |
OutputRefs lists the names ( |
parse |
string |
(optional) Parse enables parsing of log entries into structured logs |
.status
Description
ClusterLogForwarderStatus defines the observed state of ClusterLogForwarder
Type
-
object
| Property | Type | Description |
|---|---|---|
conditions |
object |
Conditions of the log forwarder. |
inputs |
Conditions |
Inputs maps input name to condition of the input. |
outputs |
Conditions |
Outputs maps output name to condition of the output. |
pipelines |
Conditions |
Pipelines maps pipeline name to condition of the pipeline. |
.spec
Description
ClusterLoggingSpec defines the desired state of ClusterLogging
Type
-
object
| Property | Type | Description |
|---|---|---|
collection |
object |
Specification of the Collection component for the cluster |
curation |
object |
(DePReCATeD) (optional) Deprecated. Specification of the Curation component for the cluster |
forwarder |
object |
(DePReCATeD) (optional) Deprecated. Specification for Forwarder component for the cluster |
logStore |
object |
(optional) Specification of the Log Storage component for the cluster |
managementState |
string |
(optional) Indicator if the resource is 'Managed' or 'Unmanaged' by the operator |
visualization |
object |
(optional) Specification of the Visualization component for the cluster |
.spec.collection
Description
This is the struct that will contain information pertinent to Log and event collection
Type
-
object
| Property | Type | Description |
|---|---|---|
resources |
object |
(optional) The resource requirements for the collector |
nodeSelector |
object |
(optional) Define which Nodes the Pods are scheduled on. |
tolerations |
array |
(optional) Define the tolerations the Pods will accept |
fluentd |
object |
(optional) Fluentd represents the configuration for forwarders of type fluentd. |
logs |
object |
(DePReCATeD) (optional) Deprecated. Specification of Log Collection for the cluster |
type |
string |
(optional) The type of Log Collection to configure |
.spec.collection.fluentd.buffer
Description
FluentdBufferSpec represents a subset of fluentd buffer parameters to tune the buffer configuration for all fluentd outputs. It supports a subset of parameters to configure buffer and queue sizing, flush operations and retry flushing.
For general parameters refer to: https://docs.fluentd.org/configuration/buffer-section#buffering-parameters
For flush parameters refer to: https://docs.fluentd.org/configuration/buffer-section#flushing-parameters
For retry parameters refer to: https://docs.fluentd.org/configuration/buffer-section#retries-parameters
Type
-
object
| Property | Type | Description |
|---|---|---|
chunkLimitSize |
string |
(optional) ChunkLimitSize represents the maximum size of each chunk. events will be |
flushInterval |
string |
(optional) FlushInterval represents the time duration to wait between two consecutive flush |
flushMode |
string |
(optional) FlushMode represents the mode of the flushing thread to write chunks. The mode |
flushThreadCount |
int |
(optional) FlushThreadCount reprents the number of threads used by the fluentd buffer |
overflowAction |
string |
(optional) OverflowAction represents the action for the fluentd buffer plugin to |
retryMaxInterval |
string |
(optional) RetryMaxInterval represents the maximum time interval for exponential backoff |
retryTimeout |
string |
(optional) RetryTimeout represents the maximum time interval to attempt retries before giving up |
retryType |
string |
(optional) RetryType represents the type of retrying flush operations. Flush operations can |
retryWait |
string |
(optional) RetryWait represents the time duration between two consecutive retries to flush |
totalLimitSize |
string |
(optional) TotalLimitSize represents the threshold of node space allowed per fluentd |
.spec.collection.fluentd.inFile
Description
FluentdInFileSpec represents a subset of fluentd in-tail plugin parameters to tune the configuration for all fluentd in-tail inputs.
For general parameters refer to: https://docs.fluentd.org/input/tail#parameters
.spec.collection.logs.fluentd
.spec.collection.logs.fluentd.tolerations[]
Description
Type
-
array
| Property | Type | Description |
|---|---|---|
effect |
string |
(optional) effect indicates the taint effect to match. empty means match all taint effects. |
key |
string |
(optional) Key is the taint key that the toleration applies to. empty means match all taint keys. |
operator |
string |
(optional) Operator represents a key's relationship to the value. |
tolerationSeconds |
int |
(optional) TolerationSeconds represents the period of time the toleration (which must be |
value |
string |
(optional) Value is the taint value the toleration matches to. |
.spec.curation.curator.tolerations[]
Description
Type
-
array
| Property | Type | Description |
|---|---|---|
effect |
string |
(optional) effect indicates the taint effect to match. empty means match all taint effects. |
key |
string |
(optional) Key is the taint key that the toleration applies to. empty means match all taint keys. |
operator |
string |
(optional) Operator represents a key's relationship to the value. |
tolerationSeconds |
int |
(optional) TolerationSeconds represents the period of time the toleration (which must be |
value |
string |
(optional) Value is the taint value the toleration matches to. |
.spec.forwarder
.spec.forwarder.fluentd.buffer
Description
FluentdBufferSpec represents a subset of fluentd buffer parameters to tune the buffer configuration for all fluentd outputs. It supports a subset of parameters to configure buffer and queue sizing, flush operations and retry flushing.
For general parameters refer to: https://docs.fluentd.org/configuration/buffer-section#buffering-parameters
For flush parameters refer to: https://docs.fluentd.org/configuration/buffer-section#flushing-parameters
For retry parameters refer to: https://docs.fluentd.org/configuration/buffer-section#retries-parameters
Type
-
object
| Property | Type | Description |
|---|---|---|
chunkLimitSize |
string |
(optional) ChunkLimitSize represents the maximum size of each chunk. events will be |
flushInterval |
string |
(optional) FlushInterval represents the time duration to wait between two consecutive flush |
flushMode |
string |
(optional) FlushMode represents the mode of the flushing thread to write chunks. The mode |
flushThreadCount |
int |
(optional) FlushThreadCount reprents the number of threads used by the fluentd buffer |
overflowAction |
string |
(optional) OverflowAction represents the action for the fluentd buffer plugin to |
retryMaxInterval |
string |
(optional) RetryMaxInterval represents the maximum time interval for exponential backoff |
retryTimeout |
string |
(optional) RetryTimeout represents the maximum time interval to attempt retries before giving up |
retryType |
string |
(optional) RetryType represents the type of retrying flush operations. Flush operations can |
retryWait |
string |
(optional) RetryWait represents the time duration between two consecutive retries to flush |
totalLimitSize |
string |
(optional) TotalLimitSize represents the threshold of node space allowed per fluentd |
.spec.forwarder.fluentd.inFile
Description
FluentdInFileSpec represents a subset of fluentd in-tail plugin parameters to tune the configuration for all fluentd in-tail inputs.
For general parameters refer to: https://docs.fluentd.org/input/tail#parameters
.spec.logStore
Description
The LogStoreSpec contains information about how logs are stored.
Type
-
object
| Property | Type | Description |
|---|---|---|
elasticsearch |
object |
Specification of the elasticsearch Log Store component |
lokistack |
object |
LokiStack contains information about which LokiStack to use for log storage if Type is set to LogStoreTypeLokiStack. |
retentionPolicy |
object |
(optional) Retention policy defines the maximum age for an index after which it should be deleted |
type |
string |
The Type of Log Storage to configure. The operator currently supports either using elasticSearch |
.spec.logStore.elasticsearch
Description
Type
-
object
| Property | Type | Description |
|---|---|---|
nodeCount |
int |
Number of nodes to deploy for elasticsearch |
nodeSelector |
object |
Define which Nodes the Pods are scheduled on. |
proxy |
object |
Specification of the elasticsearch Proxy component |
redundancyPolicy |
string |
(optional) |
resources |
object |
(optional) The resource requirements for elasticsearch |
storage |
object |
(optional) The storage specification for elasticsearch data nodes |
tolerations |
array |
.spec.logStore.elasticsearch.storage.size
Description
Type
-
object
| Property | Type | Description |
|---|---|---|
Format |
string |
Change Format at will. See the comment for Canonicalize for |
d |
object |
d is the quantity in inf.Dec form if d.Dec != nil |
i |
int |
i is the quantity in int64 scaled form, if d.Dec == nil |
s |
string |
s is the generated value of this quantity to avoid recalculation |
.spec.logStore.elasticsearch.tolerations[]
Description
Type
-
array
| Property | Type | Description |
|---|---|---|
effect |
string |
(optional) effect indicates the taint effect to match. empty means match all taint effects. |
key |
string |
(optional) Key is the taint key that the toleration applies to. empty means match all taint keys. |
operator |
string |
(optional) Operator represents a key's relationship to the value. |
tolerationSeconds |
int |
(optional) TolerationSeconds represents the period of time the toleration (which must be |
value |
string |
(optional) Value is the taint value the toleration matches to. |
.spec.logStore.retentionPolicy.application
Description
Type
-
object
| Property | Type | Description |
|---|---|---|
diskThresholdPercent |
int |
(optional) The threshold percentage of eS disk usage that when reached, old indices should be deleted (e.g. 75) |
maxAge |
string |
(optional) |
namespaceSpec |
array |
(optional) The per namespace specification to delete documents older than a given minimum age |
pruneNamespacesInterval |
string |
(optional) How often to run a new prune-namespaces job |
.spec.logStore.retentionPolicy.audit
Description
Type
-
object
| Property | Type | Description |
|---|---|---|
diskThresholdPercent |
int |
(optional) The threshold percentage of eS disk usage that when reached, old indices should be deleted (e.g. 75) |
maxAge |
string |
(optional) |
namespaceSpec |
array |
(optional) The per namespace specification to delete documents older than a given minimum age |
pruneNamespacesInterval |
string |
(optional) How often to run a new prune-namespaces job |
.spec.logStore.retentionPolicy.infra
Description
Type
-
object
| Property | Type | Description |
|---|---|---|
diskThresholdPercent |
int |
(optional) The threshold percentage of eS disk usage that when reached, old indices should be deleted (e.g. 75) |
maxAge |
string |
(optional) |
namespaceSpec |
array |
(optional) The per namespace specification to delete documents older than a given minimum age |
pruneNamespacesInterval |
string |
(optional) How often to run a new prune-namespaces job |
.spec.visualization.kibana
Description
Type
-
object
| Property | Type | Description |
|---|---|---|
nodeSelector |
object |
Define which Nodes the Pods are scheduled on. |
proxy |
object |
Specification of the Kibana Proxy component |
replicas |
int |
Number of instances to deploy for a Kibana deployment |
resources |
object |
(optional) The resource requirements for Kibana |
tolerations |
array |
.spec.visualization.kibana.tolerations[]
Description
Type
-
array
| Property | Type | Description |
|---|---|---|
effect |
string |
(optional) effect indicates the taint effect to match. empty means match all taint effects. |
key |
string |
(optional) Key is the taint key that the toleration applies to. empty means match all taint keys. |
operator |
string |
(optional) Operator represents a key's relationship to the value. |
tolerationSeconds |
int |
(optional) TolerationSeconds represents the period of time the toleration (which must be |
value |
string |
(optional) Value is the taint value the toleration matches to. |
.status.logStore.elasticsearchStatus[]
Description
Type
-
array
| Property | Type | Description |
|---|---|---|
cluster |
object |
(optional) |
clusterConditions |
object |
(optional) |
clusterHealth |
string |
(optional) |
clusterName |
string |
(optional) |
deployments |
array |
(optional) |
nodeConditions |
object |
(optional) |
nodeCount |
int |
(optional) |
pods |
object |
(optional) |
replicaSets |
array |
(optional) |
shardAllocationenabled |
string |
(optional) |
statefulSets |
array |
(optional) |
.status.logStore.elasticsearchStatus[].cluster
Description
Type
-
object
| Property | Type | Description |
|---|---|---|
activePrimaryShards |
int |
The number of Active Primary Shards for the elasticsearch Cluster |
activeShards |
int |
The number of Active Shards for the elasticsearch Cluster |
initializingShards |
int |
The number of Initializing Shards for the elasticsearch Cluster |
numDataNodes |
int |
The number of Data Nodes for the elasticsearch Cluster |
numNodes |
int |
The number of Nodes for the elasticsearch Cluster |
pendingTasks |
int |
|
relocatingShards |
int |
The number of Relocating Shards for the elasticsearch Cluster |
status |
string |
The current Status of the elasticsearch Cluster |
unassignedShards |
int |
The number of Unassigned Shards for the elasticsearch Cluster |