05 May 2014

Config log4net send log to elasticsearch with fluentd and kibana - realtime and centralization.






This post will help you to config Apache log4net Library to output log to a centralization logging system, which can be used to watch realtime log events + searching + analysis ...

I will use 2 machines in this scenario :

- The Log4net machine with IP = 10.90.7.194, this machine act as the source log generator (installed or used with some application which has implemented Apache Log4net Lib). The Log4net will be configed to send log stream using syslog format on UDP protocol.

- The logging machine with IP = 10.90.7.195, this machine will recieve log stream from the upper machine. In this machine, these softwares will be installed :
  1. td-agent ( http://fluentd.org ) : act as log transmitter / receiver / parser - this app will listen and receive log stream from the generator by using in_syslog module ( http://docs.fluentd.org/articles/in_syslog ).
  2. Elasticsearch ( http://www.elasticsearch.org ): Act as a SearchEngine / Indexer, this will store log events and make they searchabled with super fast speed.
  3. Kibana ( http://www.elasticsearch.org/overview/kibana ) : A web tool connects to ElasticSearch to show fantastic dashboard.
At the final, we will have some dashboards like these :

- Events happen in realtime, with Search form, Filter, Terms by count or by max / min, ...


- List of events in realtime :


- Events in detail with message field supports multi-lines exception :



Before continuing, I strongly recommend you to have a look on these parts to take references. It will 
help you to understand how td-agent / elasticsearch / kibana working, and how to config them as we need. 

From now on, I assume that you have got cleared with them and I wont talk too much in detail, just focus on the main configuration.

>> Next to part 2 >>