Troubleshooting JVM memory issues can be daunting, but you if you don’t fix it, it can kill your application and possibly your face in front of your customers. There is good news though. With any modern APM tool, you virtually get an X-Ray vision to this labyrinth named Java Heap.
In this blog post, I’ll show you how to use AppDynamics to troubleshoot Java memory issues.
AppDynamics is one of the leading APM vendors in the market, providing tons of useful tools ranging from Hardware monitoring to End user experience monitoring to operational data analytics (Actually it can be quite overwhelming. If you don’t know what exactly you are looking for in the tool, you can go in circles. But don’t worry, I can help).
There are three major areas where AppDynamics can help in regards to Java memory troubleshooting.
We live in the world of APMs where traffic lights in fancy dashboards show us what’s wrong with our applications in a heartbeat. Definitely a proud achievement (and I highly recommend investing in an APM solution). But at times, you have to get down to very basic, old fashioned troubleshooting. And to be honest with you, it is outright fun.
In this article I want to show you how you can troubleshoot using web server access log. I am going to use Apache 2.X as example web server (Even with Nginx on the rise, apache is still the most popular web server in the world. Sorry IIS).
Before we dive in to the access log, here is a very high level architecture that shows where apache typically sits. For simplicity sake, I have not shown things like DMZ zone, Caching servers, Web Access Firewalls, Reverse proxys etc)
Analyzing logs can be fun, tricky, frustrating and valuable – all at the same time. As a problem solver, you must equip yourself with efficient tools to do the mundane work. In this article, let me show you three somewhat less popular log analysis tools. They are less popular because they are sparingly used by companies here and there (mainly due to Administrators becoming familiar with a certain tool over time). Check these out, who knows you might end up liking one of these tools and put it to good use.
Apache log4j is the foundation for java based applications. Chanisaw was written to provide a graphical view of log4j logs.
Image source: http://logging.apache.org/chainsaw/
Some notable features:
You can use expression based filtering and also do some quick-and-dirty filtering
Specify your own rules to highlight log records
Capturing remote events
Using the ‘Receiver’ concept you can configure chainsaw to capture logs from a remote source
Java memory leaks can be deadly, and difficult to troubleshoot. Are you one of those shops where you restart your Application Servers at regular intervals (weekly, daily or more frequently)? It is pathetic, is it not? Wait a minute, gone are the days where we had 128 MB memory on servers. We have several giga bytes of memory on servers, don’t we? Why do we still run into memory issues? Good question. But sad truth is there are several reasons why Memory leak is not something that will go away. All you can do is to prepare yourself. And that’s what this article is about. Let’s dive into 10 things you need to know about Java memory leak.
1. Java Heap Memory leak is different from Native Memory leak
Java heap is where the objects created by your application live. The maximum heap is determined by the –Xmx flag of the Java command line that starts the application. If you write code that leaks memory, there is where it will blow up.
Custom Dashboard is one of the most powerful features of AppDynamics. It lets you group monitoring metrics that are relevant (and make sense to the user) in one central dashboard. Custom Dashboards are ‘read only’ elements. So, you don’t have to worry about users updating any Appdynamics configuration. You can build sophisticated dashboards with drill down capabilities.
To create a new custom dashboard, click on the icon at the top and choose ‘Custom Dashboards’ as shown below
In the resulting screen, click on ‘Create Dashboard’
Agent is a vital part of Appdynamics framework. It is the agent that acts like a workhorse to pull metrics from the Application and push it to the Controller. Agent is a piece of software that is installed ON your application. The only function of the agent is to pull the monitoring metrics and send them to the AppDynamics Controller where the data is crunched and made available via the Controller UI. Note that there is NO need of a code change in your application. There is however a setup/configuration change required depending on the platform. In this article, I will explain how to instrument a Java Application.
A log file is the single most important piece of resource you need in order to tackle almost any problem with your application. I still remember having to troubleshoot complex application performance issues when APM tools were not yet born. All I had were access.log and error.log from a Web Server, standard out and standard error file from the application, and the syslog from the host OS. And guess what? They were more than enough to see what was going on.
But gone are the good old days. The complexity software and hardware infrastructure on which applications are presently deployed is beyond imagination. Application infrastructure is increasingly becoming sort of ‘black box’, and having the right tools to gain insight to this black box is mission critical.
Two parallel set of management software have emerged:
What good is an APM solution without a solid Alerting system that is easy to configure? Appdynamics alerting system is very robust with lots of options to customize.
First of all, AppDynamics comes with several pre built alerts with thresholds that are dynamically learned. This means without configuring anything, you get out-of-the-box alerting. This default alerting is visible in the Dashboard via the coloring scheme (for example, a node icon turning RED). However, the default alerting is not enough for most cases. You would want to customize the default ones, and perhaps add your own.
Two major components of Appdynamics Alerting system
Appdynamics controller UI is very versatile, responsive and easy to use. But at times, it might look like you are beating around the bush but not getting to what you really want. For example, you can easily get lost in the amount of transaction snapshots available to you. (All you have to do is use the ‘search’ box at the top right to filter out the Transaction snapshots). In this tutorial, I’ll show you the most frequently used parts of the Controller UI.
The opening Screen
The Flow map (the mighty Dashboard that you don’t have to build)
So, what can an APM tool buy you? Setting aside the hypothetical ‘Peace of mind’ marketing pitch, let me show you how exactly an APM tool can help you support your Application effectively
1. Historic Monitoring of Key Metrics
APM tool can record the monitoring metrics which are invaluable in troubleshooting. For example, take a look at the ‘response time’ graph of a particular application. You can readily see that the application suffers during business hours.