Using ELK to add and analze the logs of your application
Logging is a mechanism that developers use to record important events, errors, important and helpful information in a physical file. This is a good development practice that is followed by professional developers when writing complex applications. There are so many benefits if a good logging system is integrated into a software system.
Typically, developers may include whatever the information that is needed for the above purposes. But there are some considerations.
There are so many tools available for log implementations just one google search away. ELK is one technology stack which helps you to pipe, store, and analyze data. ELK stands for three tools: Elasticsearch, Logstash and Kibana.
Usage if the ELK stack gives the following benefits.
Logstash is a service deployed in the same server as the Telzee application runs. The logs created from the Telzee application are monitored and the changes are pushed to the Elasticsearch server.
Logstash can filter data and pinpoint which logs can not match the given filter type. Grok is a helpful Logstash plugin which helps keep the log formatting.
grok {
patterns_dir => ["/home/ubuntu"]
match => { "message" => '\[%{DATA:timestamp}\] \[%{WORD:logLevel}\] - \[%{WORD:room}\] \[%{IP:ipaddress}\] %{GREEDYDATA:content}'}
break_on_match => false}
Logstash can read logs in a defined path and push them to a number of services and Elasticsearch is one of them. To read an input log file we can define where it is located along with some other parameters.
file
{ type => "errors"
path => "/logpath/errors.log"
start_position => "beginning"
codec => multiline {
pattern => "^\[%{DATA:timestamp}\]"
negate => true
what => "previous" }}
The content of the file will go through the Grok filter and is pushed in to the output destination.
output{
elasticsearch {
hosts => ["ip:port"] }
stdout {}
}
Elasticsearch is the storage for the log files which can handle huge data chunks. Elasticsearch will also index the data if they are properly formatted so the searching process is made easy. Elasticsearch can also accept data input from other services apart from Logstash.
Kibana is a tool that offers visualization features for data stored in Elasticsearch. Users have the following benefits from the Kibana tool.