Adding Elastichsearch filebeat to Docker images

One of the projects I’m working on uses a micro-service architecture. Every micro-service is shipped as a Docker image and executed on Amazon AWS using ECS. This basically means that on a random number of servers, a (somewhat) random number of instances of a single micro-service can run … and this about 15 times (the number of micro-services we have).

In the best case this means that 15 Docker containers are running on 4 EC2 instances and thus there are 15 logs files to gather. To handle this we use the Elastichsearch ELK stack using ElasticSearch, Logstash and Kibana. Our micro-services do not directly connect to the Logstash server, instead we use filebeat to read the logfile and send it to Logstash for parsing (as such, the load of processing the logs is moved to the Logstash server).

Filebeat configuration

Filebeat is configured using YAML files, the following is a basic configuration which uses secured connection to logstash (using certificates). Using the fields property we can injection additional parameters like the environment and the application (in this case the micro-service’s name). The multi-line pattern will make sure that stacktraces are sent as one line to the server.

In this case the filebeat server will monitor the /tmp/application*.log files (which is where we log or logs)

filebeat.prospectors:
- input_type: log
  paths:
    - /tmp/application*.log
  document_type: logback  
  multiline.pattern: ^\d\d\d\d
  multiline.negate: true
  multiline.match: after
fields:
  env: %ENVIR%
  app: %APP%
output.logstash:
  hosts: ["%STASH%"]
  ssl.certificate_authorities: ["/app/ca.pem"]
  ssl.certificate: "/app/cert.pem"
  ssl.key: "/app/key.pem"

 

The %ENVIR%, %APP% and %STASH% parts will be replace later so that we can customize the logging for each environment and micro-service.

Installing filebeat in a Docker image

This is actually pretty straight forward. The following Dockerfile will install the filebeat service and on startup of the container it will run entrypoint.sh with the command of your likings.

FROM openjdk:8u111-jdk
#
#
# Stuff to get the micro-service running (this is not important for this blog)
#
#

# Monitoring
RUN wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.2.0-amd64.deb
RUN dpkg -i filebeat-5.2.0-amd64.deb
RUN rm filebeat-5.2.0-amd64.deb

COPY filebeat.yml /etc/filebeat/filebeat.yml
COPY entrypoint.sh /bin/entrypoint.sh
RUN chmod +x /bin/entrypoint.sh

ENV APP unk
ENV ENVIR unk
ENV STASH your-server:5044
#because we have a secured connection to logstash
COPY ca.pem /app/ca.pem
COPY cert.pem /app/cert.pem
COPY key.pem /app/key.pem

CMD /bin/entrypoint.sh "java -jar /app/service.jar -server --spring.profiles.active=${PROFILE}"

A couple of notes: APP and ENVIR are configure to ‘unk’ here because it’s a base image for other micro-services. Each micro-service will have it’s own Dockerfile which extends this image where it configures the APP variable. The ENVIR variable will be set during startup of the container by passing it as a variable.

Starting the service

Because Docker images don’t contain running service, we need to start the service, that’s why we need the entrypoint.sh file. This script will also alter the filebeat.yml file so that it’s configured with information about the environment and the micro-service.

#!/usr/bin/env bash
sed -i -e s/%STASH%/$STASH/g /etc/filebeat/filebeat.yml
sed -i -e s/%APP%/$APP/g /etc/filebeat/filebeat.yml
sed -i -e s/%ENVIR%/$ENVIR/g /etc/filebeat/filebeat.yml
service filebeat start
echo "$*"
/bin/sh -c "$*"