I will use 2 machines in this scenario :
- The Log4net machine with IP = 10.90.7.194, this machine act as the source log generator (installed or used with some application which has implemented Apache Log4net Lib). The Log4net will be configed to send log stream using syslog format on UDP protocol.
- The logging machine with IP = 10.90.7.195, this machine will recieve log stream from the upper machine. In this machine, these softwares will be installed :
- td-agent ( http://fluentd.org ) : act as log transmitter / receiver / parser - this app will listen and receive log stream from the generator by using in_syslog module ( http://docs.fluentd.org/articles/in_syslog ).
- Elasticsearch ( http://www.elasticsearch.org ): Act as a SearchEngine / Indexer, this will store log events and make they searchabled with super fast speed.
- Kibana ( http://www.elasticsearch.org/overview/kibana ) : A web tool connects to ElasticSearch to show fantastic dashboard.
At the final, we will have some dashboards like these :
- Events happen in realtime, with Search form, Filter, Terms by count or by max / min, ...
- Events in detail with message field supports multi-lines exception :
Before continuing, I strongly recommend you to have a look on these parts to take references. It will
help you to understand how td-agent / elasticsearch / kibana working, and how to config them as we need.
From now on, I assume that you have got cleared with them and I wont talk too much in detail, just focus on the main configuration.