23 January 2014

Postfix log centralize and analysis in realtime with fluentd tdagent elasticsearch and kibana

Preface

This tutorial will walk you through how to build a Mail Log Centralized system with Postfix, Fluentd, Elasticsearch and Kibana.

At the result, you will able to see log events happening in realtime, detail of an log record, do some analysis like Top Senders, Top Receivers, Top Status ...

Some screenshots :

Events happening in realtime.

Detail of an log event.

Term analytic as Top Senders, Top Receivers, Top Relays ...

List of events

How will all these thing work ? 

In this setup, I will use 2 servers for easy and simple :

- [Server1 - 10.90.7.194 : running Postfix as SMTP server - This act as the source log generator]
- [Server2 - 10.90.7.195 : running Fluentd (td-agent) as log receiver/parser + ElasticSearch as SearchEngine + Kibana as Web GUI Front-end]

By default, Postfix uses syslog or rsyslog to log mail event to - /var/log/maillog, you can check this in /etc/[r]syslog.conf file. When a message sent or received through postfix, it will be processed by some postfix services as smtpd, qmgr, cleanup, virtual, bounce,  ... There will be some lines in log file with the same queueID (for ex C30FDF40001) related to a message.

We don't really need all of these lines, just focus on the line which includes the rcpt-to address (mail-to), status of the message. For examples :

Jan 23 16:21:48 ServerName postfix/smtp[26408]: 4367111B8004: to=<someone@gmail.com>, relay=gmail-smtp-in.l.google.com[74.125.25.27]:25, delay=2.4, delays=0.1/0/1.2/1
.1, dsn=2.0.0, status=sent (250 2.0.0 OK 1390468929 fu1si13358750pbc.104 - gsmtp)

1. Config rsyslog to filter the main log line

On the Server1 which is running postfix, we will use the feature Property-Based Filters of rsyslog to filter this type of message. We will filter the messages (log line) which contains string : "status=" and then transmit these messages to the [Server2] by UDP at port 5140.

This config works only with rsyslog and not syslog, so you need to install rsyslog and stop the syslog service.

# yum install rsyslog
# /etc/init.d/syslog stop 

Config the rsyslog.conf to filter and transmit "status=" message. # vim /etc/rsyslog.conf 
Insert this line :

:msg, contains, "status=" @10.90.7.195:5140

then restart the rsyslog service : # /etc/init.d/rsyslog restart

From now on, rsyslog besides writing log to /var/log/maillog file - it also send another log stream to the server2 at UDP port 5140.

Continue to part 2 >> Config td-agent