Collecting Logs with AWS Distro for OpenTelemetry: A Practical Guide

Hello there.
How are you doing?
I'm Ryo Yoshii, a huge fan of the phrase "No human labor is no human error."

AWS Distro for OpenTelemetry (ADOT) now supports logs.
You can collect logs using the ADOT Collector or OpenTelemetry SDKs (Java, JavaScript, .NET, Python) and forward them to any backend that supports OTLP.

Some of you might be thinking, "I already have Fluent Bit, so I'm fine." Well, please don't be so quick—take a look at this update.
If you are already using the ADOT Collector or OTEL SDKs to collect traces and metrics, you might find it beneficial to collect logs the same way.

Also, if you can attach the same OTEL metadata used for traces and metrics to your logs, you can correlate them, which could be very useful for analysis.

However, please note that the stability is currently in beta. Be aware that there may be breaking changes.
AWS CloudWatch Logs Exporter

Trying it out

I set up an application container and a sidecar (ADOT Collector) on ECS to forward log files written to /var/log/logFile.log to CloudWatch Logs.
I chose CloudWatch Logs because it's the easiest, but any backend that supports OTLP will work. (I know there isn't much point in forwarding to CloudWatch Logs specifically, but please, no comments on that!)

ECS

I tested this using ECS on Fargate.

The ECS task definition is shown below. I have included only the necessary parts; please configure the omitted items as appropriate.
The key point is defining a task storage volume in volumes and mounting it in both containers. This allows the OTEL Collector to read the log files generated by the application.

{
    "containerDefinitions": [
        {
            "name": "aws-otel-collector",
            "image": "public.ecr.aws/aws-observability/aws-otel-collector:v0.35.0",
            "mountPoints": [
                {
                    "sourceVolume": "varlog",
                    "containerPath": "/var/log"
                }
            ],
            "secrets": [
                {
                    "name": "AOT_CONFIG_CONTENT",
                    "valueFrom": "adotconfig"
                }
            ]
        },
        {
            "name": "your_app",
            "image": "your_image:tag",
            "mountPoints": [
                {
                    "sourceVolume": "varlog",
                    "containerPath": "/var/log"
                }
            ]
        }
    ],
    "volumes": [
        {
            "name": "varlog",
            "host": {}
        }
    ]
}

SSM Parameter Store

I store the ADOT Collector configuration in the SSM Parameter Store.
The filelog receiver is set to pick up logs from /var/log/*.log.
The AWS CloudWatch Logs Exporter is used to specify the log group and other settings.

receivers:
  filelog:
    include: [ /var/log/*.log ]

processors:
  batch:

  memory_limiter:
    limit_mib: 100
    check_interval: 5s


exporters:
  awscloudwatchlogs:
    log_group_name: "/ecs/your_app"
    log_stream_name: "logfile"
    region: "us-west-2"
    log_retention: 7

service:
  pipelines:
    logs:
      receivers: [filelog]
      processors: [memory_limiter,batch]
      exporters: [awscloudwatchlogs]

Application Logs

All that's left is for the application to output log files to /var/log/.
Log rotation should be implemented on the application side.

Conclusion

I tried out the log support in ADOT.
While you may have already achieved log forwarding using other mechanisms, there are cases where it's great to be able to collect the basic three—logs, traces, and metrics—using only the ADOT Collector.

References

Container Logs Collector Configuration
AWS CloudWatch Logs Exporter
Bind Mounts