Create a Central Logging System

-
Logging is the most important part of any system. They give you insights about the application, what kind of errors are occurring and what components are causing the errors and also about how the application is reacting and working when something wrong happens. The usual practice is almost every application write to logs either in the file system or in Database. If your application is running on multiple hosts then designing a central logging system becomes crucial since you can collect, aggregate and maintain logs at one centralized location and do operations on top of them.
There are many tools available to which can solve some part of the problem but we need to build a robust application using all these tools.

                                                                         
Create a Centralised Logging Application

Collection

Imagine you have a web application running on the server and if something goes down, your developers or operations team need to access log data quickly in order to troubleshoot live issues, you would need a solution which can monitor the changes in the log files in almost real-time. Hence you would need to replicate the logs.

Transport

If you have multiple hosts running then logs data can accumulate quickly. There should be an efficient and reliable way to transport this data to the centralized application and ensure data is not lost. There are many frameworks available to transport log data. One way is directly plug input sources and framework can start collecting logs and another way is to send log data via API, application code is written to log directly to these sources it reduces latency and improves reliability.
If you want to provide a number of input sources you can use Logstash — Open Source Log collector, written in Ruby.

Storage

Now we have transport in place, logs will need a destination, a storage where all the log data will be saved. The system should be highly scalable as the data will keep on growing and it should be able to handle the growth over time. Logs data will depend on the how huge your applications are if your application is running on multiple servers or in many containers it will generate more logs. ElasticSearch or HDFS is a good choice for interactive data analysis and working with raw data more effectively.

Analysis

Logs are meant for analysis and analytics. Once your logs are stored in a centralized location, you need a way to analyze them. There are many tools available for log analysis, if you need a UI for analysis, you can parse all the data in ElasticSearch and use Kibana or Greylog2 to query and inspect the data. Grafana and Kibana can be used to show real-time data analytics.

Alerting

It’s nice to have an alerting system which will alert us to any change in the log patterns or calculated metrics. Logs are very useful for troubleshooting errors. It’s far better to have some alerting build in the logging application system which will send an email or notify us then to have someone keep watching logs for any changes. There are many error reporting tools available, you can use Sentry or Honeybadger.

Thanks for dropping by!! Feel free to comment on this post or you can drop me an email at naik899@gmail.com

Comments

Post a Comment

Popular posts from this blog

Serve GZip CSS and JS from AWS S3

-
Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance. This means customers of all sizes and industries can use it to store and protect any amount of data for a range of use cases, such as websites, mobile applications, backup and restore, archive, enterprise applications, IoT devices, and big data analytics. If you are looking to gzip  your static JS and CSS when hosting on S3, however, it’s not obvious how to do so, and results often bring up other AWS hosting services as well. Here’s the quick and simple way to serve gzipped JS and CSS from AWS S3. S3 needs those files compressed before they are uploaded, it can’t do the actual gzipping for us. This is done with a simple command, which I recommend be added as a script to your build step, or your continuous integration process. gzip -9 /filepath/bootstrap.js or gzip -9 /filepath/bootstrap.css The -9 denotes that we want the
Read more

Create a calendar invite (ics file) in ASP NET

-
Creating a calendar invite (ics file) and attaching a email via service or sending an email with downloadable calendar invite is what we are going to learn in following posts. In this post, I will show how you can use ICal library from Nuget and create a calendar invite. The calendar begins with the keyword  BEGIN:VCALENDAR  and ends with  END:VCALENDAR . In between the calendar you can add as many events as you want beginning each event with  BEGIN:VEVENT  and ending with  END:VEVENT . There is a catch when adding multiple events to a calendar, and that is the type of METHOD you use. With METHOD:PUBLISH, you can add multiple events and upon opening the iCal with MS Outlook it will create a new calendar, separate from your main Exchange calendar. METHOD:REQUEST creates a calendar invite, so you can only include one event and upon opening with MS Outlook it will ask you to accept the event and add it to your main Exchange calendar. With this method you have to include the organizer
Read more