A log file is the single most important piece of resource you need in order to tackle almost any problem with your application. I still remember having to troubleshoot complex application performance issues when APM tools were not yet born. All I had were access.log and error.log from a Web Server, standard out and standard error file from the application, and the syslog from the host OS. And guess what? They were more than enough to see what was going on.
But gone are the good old days. The complexity software and hardware infrastructure on which applications are presently deployed is beyond imagination. Application infrastructure is increasingly becoming sort of ‘black box’, and having the right tools to gain insight to this black box is mission critical.
Two parallel set of management software have emerged:
- APM tools (Application Performance Management Tools)
- Log management tools
Let me explain what these management tools can do and more importantly, can’t do.
APM Tools
These tools are typically agent based and provide deep dive metrics from your application at code level. They also provide end user experience monitoring and somewhat limited Database and Hardware monitoring. They have powerful dishoarding capabilities and enables you to visualize the flow of the application. Interestingly, they can also monitor log files and even catch preconfigured error strings. A solid APM tool is a must if you are serious about your business. Features are being added at rapid pace as completion among vendors are growing stronger.
APM tools can:
- Monitor overall application flow – end-to-end
- Provide traces that include code level details
- Extend the monitoring capabilities using plugins
APM tools can’t:
- Act as a Log management system (i.e indexing and searching the log files)
- Provide performance metrics based on Log files (A seasoned administrator can get creative and ultimately configure an APM tool to parse out log file and make some metrics out of it. But it won’t be a reliable and scalable solution)
- Detect security attacks/violations (Again, seasoned administrator might think otherwise)
Popular APM vendors include (but not limited to): New Relic, AppDynamics, CA APM, Dynatrace
An APM Dashboard (Source: New Relic)
Log Management Tools
These tools consume and index Logs, and provide an easy to use interface to search them. They can be configured to automatically alert based on certain strings found in the log files. They can also deliver extremely insightful analytics at surprising speeds. Most of these tools use some sort of no-sql movement database (such as Apache Lucene).
Log Management tools can:
- Provide a scalable, reliable way to store and index your log files
- Enable you to search log files at great speeds
- Create insightful analytics based on the log data
- Monitor for security violations and cyber-attacks based on the log file
Log Management Tools can’t:
- Provide an end-to-end application performance monitoring
- Reveal code level traces
Popular Log Management Tools vendors include Elastic Search, Splunk and the open source Solr.
A Log Management tool Dashboard (Source: Elastic Search):
So, what is the verdict?
The way I see it, anyone serious about Application performance management would have two monitors in their desks – one showing the APM Dashboard and another the Log Management Dashboard. These tools complement each other and provide valuable information. It is important to understand that some of the capabilities of these tools may overlap, but fundamentally they both address different aspects of Application Performance Management. In a typical troubleshooting scenario, I will first glance at my APM Dashboard, dig into Transaction traces if need to, and then hop on to my log analysis dashboard to start searching. They both are important and required.
If you haven’t yet deployed an APM solution and a Log management solution, there are several open source products and free versions of commercial products. Don’t wait any longer. Star implementing these tools and make things happen.
Happy Monitoring !!