WebLogic

In your Application support/Developer Journey, you will definitely come across a time where you will want to really dive into the Java heap and see what is filling up the memory. We are literally talking about all those java objects that are in the heap. Mind you this can run into hundreds of millions in numbers. How do you get an insight into what is in the Heap? By analyzing a Heap dump. And how do you analyze the heap dump? By using the all-powerful Eclipse Memory Analyzer (MAT)

Before we begin using MAT, how do you capture heap dump?

Now, this varies from application to application. For example, if you are running IBM WebSphere, you could use a wsadmin script to invoke heap dump. Earlier you saw visualvm can create a heap dump for you through the visualvm interface. You can also use the command ‘jmap’ to create heap dump. Further you can configure the Application server to automatically perform a heap dump when an ‘out of memory’ error is encountered (XX:-HeapDumpOnOutOfMemoryError). Note that this option is verified to work on Oracle Hotspot JVM only.

Downloading Eclipse MAT:

Read More

IBM has given us a great tool that we can use to analyze verbose GC files literally in seconds. I’m not kidding. All you have to do is generate verbose GC logs for few hours and feed the file to this tool and Bingo; it analyzes the file and tells you what you need to know. Let’s see this tool in action.

Downloading IBM Pattern Modeling and Analysis Tool for Java Garbage Collector:

https://www.ibm.com/developerworks/community/groups/service/html/communityview?communityUuid=22d56091-3a7b-4497-b36e-634b51838e11

Click on the ‘download’ link at the bottom of the page. You will get a jar file named “ga<version>.jar”.

For example, ga456.jar.

Invoking IBM PMAT:

Read More

VisualVM is a free monitoring tool for the JVM. It is much more feature rich than Jconsole. It comes built in on the Oracle JDKs. You can also download a standalone version from visualvm.java.net.

Invoking visualvm:

The tool ‘jvisualvm’ is located in bin directory under ‘JDK Home’.

Jvisualvm

Note, similar to jconsole, if you are running visualvm with the same user id as the application, the application will appear under ‘Local’ and you can begin monitoring right away. However, if that is notthe case, you must first enable JMX remote management by adding the following parameters to the Java command line.

Read More

One of the built-in tools that can be used to view the memory usage graphically is jconsole. It comes with JDK. Jconsole uses JMX (Java Management Extension) to pull several vital stats from the running Java application. While jconsole may seem little outdated (and it is) it is still valuable if you want a quick GUI to view JVM stats

How to invoke Jconsole ?

Read More

So you enabled verbose GC logs and obtained the logs. Congratulations. You have done one of the most important steps in troubleshooting JVM memory issues. But merely having the log does not do any good. You need to analyze and determine if there is a smoking gun. Here is how to do it.

1. First make sure you collect the logs for at least few hours. Longer the better.

2. Open the log file in a text editor.

3. You are primarily looking for two things

a. Is the GC happening very frequently?

b. Is each GC cycle taking long time?

Read More

If you have to choose only one tool to troubleshoot Java memory related issues, you must choose the ‘verbose gc logs’. Verbose GC log has so much information that you can diagnose almost all types of memory issues just by carefully reviewing the data in this file. It is such a valuable tool.

What’s more? The load exerted on the JVM due to verbose GC logging is virtually nothing. So, don’t skimp on enabling verbose gc logs when you need to.

What exactly do the verbose GC logs reveal?

Read More

If you think you will never run into out of Memory error because your code is leak proof and your application server is rock solid, think again. Your application can run out of what eludes most developers and Systems Administrators – Native memory.

To put it blunt, Native memory is the memory used by JVM itself for its internal low-level tasks (for example file handles) and the memory used by any native code (such as a C library within the JVM).

What are the symptoms of native memory issues?

1. Unexpected application errors

2. One of the following error message in the log file

a. java.lang.OutOfMemoryError:Detail Message: request <size> bytes for <reason>. Out of swap space?

b. java.lang.OutOfMemoryError:Detail Message: <reason> <stack trace> (Native method)

Note that the errors above are for Oracle HotSpot. The error varies by the JVM vendor.

What causes native memory issue?

Note that the JVM is just a process as far as OS is concerned and hence it is subject to all resource restrictions that the OS poses. Here are some of the major causes of native memory issues

1. Application code issue where fault code creates tons of low level elements (such as File handles,network sockets etc).

2. Third party library code issue (for example C libraries) that suffer from memory leaks

How to prevent native memory issue?

There is not a lot you can do to prevent a native memory issue as these issues are totally unexpected.

But you can try the following

1. Ensure the third party native libraries you use are correct versions and for the appropriate OS and architecture

2. Ensure the application code does not leave lingering network or file handles.

One useful way to determine if you have native memory issue is to check the ‘process size’ of the JVM viewed using operation system tools (for example ‘top’ in Unix). This process size will include Heap and native memory used.

Moving on, let us explore various troubleshooting tools available to aid you with Memory issues

Coming up: Troubleshooting GC: Using Verbose GC logs

Previous: Troubleshooting GC: Excessive Garbage Collection

GC is supposed to save the developers from the tedious task of managing the memory (remember malloc and free and how accidentally you can free an object that is actually being referenced somewhere else and thereby causing unexpected errors ??). However, the very same GC can be a resource hog and bring your application to its knees.

What are the symptoms of Excessive Garbage Collection ?

1. No apparent errors in the logs files but the application is extremely slow
2. Restarting application seems to provide relieve for short period but the slowness comes back eventually

What are the causes of excessive garbage collection ?

Read More

If there is one memory related issue that strikes you off guard, that must be ‘running out of permgen space’. Fortunately it is easy to fix.

What are the symptoms of running out of Permgen space ?

1. Unexpected application errors
2. Unable to start the application
3. java.lang.OutOfMemoryError:PermGen space in the log file.

What causes to run out of Permgen space ?

Permgen is a an area of Heap that stores class objects and interned strings. You don’t have to get into the details but know that permgen area will fill up if you load tons of classes, which will be the case if you load numerous third party libraries. Specifically, some of the ‘javaagent’ libraries (an important component in APM tools) are known to demand large permgen space.

The default permgen size (this varies by version and platform, typically 64MB) may not be enough in some cases especially if you use several third party libraries as mentioned above.

How to prevent Heap ‘running out of Permgen space’ ?

1. Try pumping up the permgen space to 256MB or more if you need

-XX:MaxPermSize=256m

2. Ensure that you are NOT loading tons of third party libraries that you don’t need. Check the classpath, java library path and startup logs to see if you find any clue.

3. Upgrade to Java 1.8. Note that there is NO permgen in java 1.8 onward. It has been replaced by Metaspace. But note that this may not be cure-all. Check out my other post: One important change in Memory Management in Java 8

Moving on, you may not see any error at all in your log files but your application could be extremely slow due to this lethal Heap related issue – Excessive garbage collection.

Coming up: Troubleshooting GC: Excessive garbage collection

Previous: Troubleshooting GC: Heap Fragmentation