<< Back to part 1 <<
2. Config Fluentd (td-agent) to receive log stream from Postfix
Fluentd (td-agent) is really a very good log transport and parser, it has a very clearly modular model, support for lot of log format - including custom format, it also has a lot of plugins which support multiple database type.Fluentd (td-agent) also supports H.A (high availability) and log stream load balancing - this will help able to scale out to very heavy traffic model.
We will install td-agent on the server2 to receive log stream from the server1. Download the rpm package from rpmse : http://www.rpmse.org/#/dashboard?s=td-agent
The best rpm search engine I've used, millions of packages have been indexed.
Td-agent also provides a yum based repo-file, just create /etc/yum.repos.d/td-agent.repo include :
[treasuredata]
name=TreasureData
baseurl=http://packages.treasure-data.com/redhat/$basearch
gpgcheck=0
then # yum install td-agent
The main config file : /etc/td-agent/td-agent.conf
Log files stay at : /var/log/td-agent/
- fluent-plugin-parser : parse event log into fields : queueid, rcpt-to, relay, status.
- fluent-plugin-elasticsearch : output log stream into ElasticSearch.
To install plugin :
# cd /usr/lib64/fluent/ruby/bin/
# ./fluent-gem install fluent-plugin-parser
# ./fluent-gem install fluent-plugin-elasticsearch
Now we need to config the td-agent service : # vim /etc/td-agent/td-agent.conf
### Listen on UDP 5140 ###
<source>
type syslog
port 5140
bind 0.0.0.0
tag syslog
</source>
### Parse events into fields : queueid, rcpt-to, relay, status ###
<match syslog.mail.info>
type parser
remove_prefix syslog
format /^(?<queueid>[^ ]*): to=<(?<rcpt-to>[^ ]*)>, relay=(?<relay>[^ ]*), [^*]* status=(?<status>[^ ]*)/
key_name message
reserve_data yes
</match>
### Write event to ElasticSearch ###
<match mail.info>
buffer_type file
buffer_path /mnt/ramdisk/postfix-mail.buff
buffer_chunk_limit 4m
buffer_queue_limit 50
flush_interval 0s
type elasticsearch
logstash_format true
logstash_prefix postfix_mail
</match>
Restart the td-agent service for applying : # /etc/init.d/td-agent restart
Check to see if the td-agent has open udp port :
# netstat -pnatu|grep 5140
and it works ! Ready to receive input log stream.
Check to see if the td-agent has open udp port :
# netstat -pnatu|grep 5140
udp 0 0 0.0.0.0:5140 0.0.0.0:* 6662/ruby
and it works ! Ready to receive input log stream.
Explain the config :
The config file itself is very clearly, but there are something that we need to notice :
The config file itself is very clearly, but there are something that we need to notice :
- Module source syslog is built-in, we dont need to install it.
- format /^(?<queueid>[^ ]*): to=<(?<rcpt-to>[^ ]*)>, relay=(?<relay>[^ ]*), [^*]* status=(?<status>[^ ]*)/ . this using regexp to parse the mail log into some fields before writing to ElasticSearch. You can try ignore this module to see the difference.
- buffer_path /mnt/ramdisk/postfix-mail.buff : I have mount a tmpfs (using ram) filesystem to /mnt/ramdisk to speedup the buffer and to able to see the buffered files changing in realtime (by default, this module does use memory to place the buffer). This is really such a performance hint, more detail here : http://docs.fluentd.org/articles/buf_file
- For debuging purpose, instead of writing to ElasticSearch, you can also try writing events log to normal files, using the out_file module : http://docs.fluentd.org/articles/out_file