Documentation Grafana Cloud Instrument and send data Grafana Alloy Reference Components otelcol otelcol.exporter.awss3

otelcol.exporter.awss3

EXPERIMENTAL: This is an experimental component. Experimental components are subject to frequent breaking changes, and may be removed with no equivalent replacement. To enable and use an experimental component, you must set the stability.level flag to experimental.

otelcol.exporter.awss3 accepts telemetry data from other otelcol components and writes them to an AWS S3 bucket.

You can specify multiple otelcol.exporter.awss3 components by giving them different labels.

Usage

alloy Copy

otelcol.exporter.awss3 "<LABEL>" {
  s3_uploader {
    region = "<AWS_REGION>"
    s3_bucket = "<S3_BUCKET_NAME>"
    s3_prefix = "<PREFIX_FOR_S3_KEY>"
  }
}

Arguments

You can use the following argument with otelcol.exporter.awss3:

Blocks

You can use the following blocks with otelcol.exporter.awss3:

s3_uploader

Required

The s3_uploader block configures the AWS S3 bucket details used by the component.

The following arguments are supported:

debug_metrics

The debug_metrics block configures the metrics that this component generates to monitor its state.

The following arguments are supported:

disable_high_cardinality_metrics is the Alloy equivalent to the telemetry.disableHighCardinalityMetrics feature gate in the OpenTelemetry Collector. It removes attributes that could cause high cardinality metrics. For example, attributes with IP addresses and port numbers in metrics about HTTP and gRPC connections are removed.

marshaler

Marshaler determines the format of data sent to AWS S3. Currently, the following marshalers are implemented:

• otlp_json (default): the OpenTelemetry Protocol format, represented as JSON.
• otlp_proto: the OpenTelemetry Protocol format, represented as Protocol Buffers. A single protobuf message is written into each object.
• sumo_ic: the Sumo Logic Installed Collector Archive format. This format is supported only for logs.
• body: export the log body as string. This format is supported only for logs.

The following arguments are supported:

sending_queue

The sending_queue block configures an in-memory buffer of batches before data is sent to S3.

The following arguments are supported:

The blocking argument is deprecated in favor of the block_on_overflow argument.

When block_on_overflow is true, the component will wait for space. Otherwise, operations will immediately return a retryable error.

When enabled is true, data is first written to an in-memory buffer before sending it to the configured server. Batches sent to the component’s input exported field are added to the buffer as long as the number of unsent batches doesn’t exceed the configured queue_size.

queue_size determines how long an endpoint outage is tolerated. Assuming 100 requests/second, the default queue size 1000 provides about 10 seconds of outage tolerance. To calculate the correct value for queue_size, multiply the average number of outgoing requests per second by the time in seconds that outages are tolerated. A very high value can cause Out Of Memory (OOM) kills.

The sizer argument could be set to:

• requests: number of incoming batches of metrics, logs, traces (the most performant option).
• items: number of the smallest parts of each signal (spans, metric data points, log records).
• bytes: the size of serialized data in bytes (the least performant option).

The num_consumers argument controls how many readers read from the buffer and send data in parallel. Larger values of num_consumers allow data to be sent more quickly at the expense of increased network traffic.

If an otelcol.storage.* component is configured and provided in the queue’s storage argument, the queue uses the provided storage extension to provide a persistent queue and the queue is no longer stored in memory. Any data persisted will be processed on startup if Alloy is killed or restarted. Refer to the exporterhelper documentation in the OpenTelemetry Collector repository for more details.

resource_attrs_to_s3 block

The following arguments are supported:

When s3_prefix is set, it dynamically overrides s3_uploader > s3_prefix. If the specified resource attribute exists in the data, its value will be used as the prefix. Otherwise, s3_uploader > s3_prefix will serve as the fallback.

Compression

• none (default): File compression isn’t used.
• gzip: Files are compressed with Gzip. This doesn’t support sumo_icmarshaler.

Exported fields

The following fields are exported and can be referenced by other components:

input accepts otelcol.Consumer data for any telemetry signal (metrics, logs, or traces).

Component health

otelcol.exporter.awss3 is only reported as unhealthy if given an invalid configuration.

Debug information

otelcol.exporter.awss3 doesn’t expose any component-specific debug information.

Debug metrics

• otelcol_exporter_queue_capacity (gauge): Fixed capacity of the retry queue (in batches).
• otelcol_exporter_queue_size (gauge): Current size of the retry queue (in batches).
• otelcol_exporter_send_failed_spans_total (counter): Number of spans in failed attempts to send to destination.
• otelcol_exporter_sent_spans_total (counter): Number of spans successfully sent to destination.
• rpc_client_duration_milliseconds (histogram): Measures the duration of inbound RPC.
• rpc_client_request_size_bytes (histogram): Measures size of RPC request messages (uncompressed).
• rpc_client_requests_per_rpc (histogram): Measures the number of messages received per RPC. Should be 1 for all non-streaming RPCs.
• rpc_client_response_size_bytes (histogram): Measures size of RPC response messages (uncompressed).
• rpc_client_responses_per_rpc (histogram): Measures the number of messages received per RPC. Should be 1 for all non-streaming RPCs.

Example

This example forwards scrape logs to an AWS S3 Bucket:

alloy Copy

local.file_match "logs" {
  path_targets = [{
    __address__ = "localhost",
    __path__    = "/var/log/{syslog,messages,*.log}",
    instance    = constants.hostname,
    job         = "integrations/node_exporter",
  }]
}

loki.source.file "logs" {
  targets    = local.file_match.logs.targets
  forward_to = [otelcol.receiver.loki.default.receiver]
}

otelcol.receiver.loki "default" {
  output {
    logs = [otelcol.exporter.awss3.logs.input]
  }
}

otelcol.exporter.awss3 "logs" {
  s3_uploader {
    region = "us-east-1"
    s3_bucket = "logs_bucket"
    s3_prefix = "logs"
  }
}

Compatible components

otelcol.exporter.awss3 has exports that can be consumed by the following components:

• Components that consume OpenTelemetry otelcol.Consumer