≡ Menu

Java Programming

Troubleshooting GC: Using Jconsole

One of the built-in tools that can be used to view the memory usage graphically is jconsole. It comes with JDK. Jconsole uses JMX (Java Management Extension) to pull several vital stats from the running Java application. While jconsole may seem little outdated (and it is) it is still valuable if you want a quick GUI to view JVM stats

How to invoke Jconsole ?

Read More

So you enabled verbose GC logs and obtained the logs. Congratulations. You have done one of the most important steps in troubleshooting JVM memory issues. But merely having the log does not do any good. You need to analyze and determine if there is a smoking gun. Here is how to do it.

1. First make sure you collect the logs for at least few hours. Longer the better.

2. Open the log file in a text editor.

3. You are primarily looking for two things

a. Is the GC happening very frequently?

b. Is each GC cycle taking long time?

Read More

Troubleshooting GC: Using Verbose GC logs

If you have to choose only one tool to troubleshoot Java memory related issues, you must choose the ‘verbose gc logs’. Verbose GC log has so much information that you can diagnose almost all types of memory issues just by carefully reviewing the data in this file. It is such a valuable tool.

What’s more? The load exerted on the JVM due to verbose GC logging is virtually nothing. So, don’t skimp on enabling verbose gc logs when you need to.

What exactly do the verbose GC logs reveal?

Read More

Troubleshooting GC: Native memory issues

If you think you will never run into out of Memory error because your code is leak proof and your application server is rock solid, think again. Your application can run out of what eludes most developers and Systems Administrators – Native memory.

To put it blunt, Native memory is the memory used by JVM itself for its internal low-level tasks (for example file handles) and the memory used by any native code (such as a C library within the JVM).

What are the symptoms of native memory issues?

1. Unexpected application errors

2. One of the following error message in the log file

a. java.lang.OutOfMemoryError:Detail Message: request <size> bytes for <reason>. Out of swap space?

b. java.lang.OutOfMemoryError:Detail Message: <reason> <stack trace> (Native method)

Note that the errors above are for Oracle HotSpot. The error varies by the JVM vendor.

What causes native memory issue?

Note that the JVM is just a process as far as OS is concerned and hence it is subject to all resource restrictions that the OS poses. Here are some of the major causes of native memory issues

1. Application code issue where fault code creates tons of low level elements (such as File handles,network sockets etc).

2. Third party library code issue (for example C libraries) that suffer from memory leaks

How to prevent native memory issue?

There is not a lot you can do to prevent a native memory issue as these issues are totally unexpected.

But you can try the following

1. Ensure the third party native libraries you use are correct versions and for the appropriate OS and architecture

2. Ensure the application code does not leave lingering network or file handles.

One useful way to determine if you have native memory issue is to check the ‘process size’ of the JVM viewed using operation system tools (for example ‘top’ in Unix). This process size will include Heap and native memory used.

Moving on, let us explore various troubleshooting tools available to aid you with Memory issues

Coming up: Troubleshooting GC: Using Verbose GC logs

Previous: Troubleshooting GC: Excessive Garbage Collection

Troubleshooting GC: Excessive garbage collection

GC is supposed to save the developers from the tedious task of managing the memory (remember malloc and free and how accidentally you can free an object that is actually being referenced somewhere else and thereby causing unexpected errors ??). However, the very same GC can be a resource hog and bring your application to its knees.

What are the symptoms of Excessive Garbage Collection ?

1. No apparent errors in the logs files but the application is extremely slow
2. Restarting application seems to provide relieve for short period but the slowness comes back eventually

What are the causes of excessive garbage collection ?

Read More

Troubleshooting GC: Running out of PermGen space

If there is one memory related issue that strikes you off guard, that must be ‘running out of permgen space’. Fortunately it is easy to fix.

What are the symptoms of running out of Permgen space ?

1. Unexpected application errors
2. Unable to start the application
3. java.lang.OutOfMemoryError:PermGen space in the log file.

What causes to run out of Permgen space ?

Permgen is a an area of Heap that stores class objects and interned strings. You don’t have to get into the details but know that permgen area will fill up if you load tons of classes, which will be the case if you load numerous third party libraries. Specifically, some of the ‘javaagent’ libraries (an important component in APM tools) are known to demand large permgen space.

The default permgen size (this varies by version and platform, typically 64MB) may not be enough in some cases especially if you use several third party libraries as mentioned above.

How to prevent Heap ‘running out of Permgen space’ ?

1. Try pumping up the permgen space to 256MB or more if you need

-XX:MaxPermSize=256m

2. Ensure that you are NOT loading tons of third party libraries that you don’t need. Check the classpath, java library path and startup logs to see if you find any clue.

3. Upgrade to Java 1.8. Note that there is NO permgen in java 1.8 onward. It has been replaced by Metaspace. But note that this may not be cure-all. Check out my other post: One important change in Memory Management in Java 8

Moving on, you may not see any error at all in your log files but your application could be extremely slow due to this lethal Heap related issue – Excessive garbage collection.

Coming up: Troubleshooting GC: Excessive garbage collection

Previous: Troubleshooting GC: Heap Fragmentation

 

Troubleshooting GC: Heap Fragmentation

OutOfMemory error can occur due to fragmented heap. This issue is not as common as the heap exhaustion but it is a deadly issue regardless.

What are the symptoms of Heap Fragmentation ?

1. Java.lang.OutOfMemory errors in the Application log file
2. Application is not responsive or extremely slow
3. Unexpected errors in user experience

What causes Heap Fragmentation ?

Read More

Troubleshooting GC: Heap Exhaustion

When your application has used up the entire allocated heap (-Xmx) and still needs more, you run into the classic OutOfMemory error. There is simply no more memory available for your application.

What are the symptoms for Heap Exhaustion?

1.    Application is not responsive or extremely slow
2.    Java.lang.OutOfMemory error in the Application log file
3.    Possible Heap Dump file if the Application Server that is running your application is configured to do so

What causes Heap Exhaustion?

Read More

Troubleshooting GC: Various GC algorithms

GC algorithms/collectors define how GC is expected to behave at run time. There are various Collectors that you can use depending on your need. Note that the collectors vary greatly by the implementation of the JVM. This document addresses Oracle Hotspot JVM.

Note: Unless you explicitly specify a GC collector in the JVM command line, JVM selects the best collector to use based on various factors such as the hardware, platform etc

1. Serial Collector

Simplest. It uses just one thread to perform the GC operation. So, it is suited for single processor machines (are there any still left ?).

Read More

Troubleshooting GC: What is GC and how it works – Part 2

Generational heap:

The heap space is primarily divided into two segments as follows:

1. Nursery or Young generation (or Eden)

2. Old or Tenured generation

(There are other segments, and they vary by the JVM vendor. But for the most part understanding the above two segments is enough to troubleshoot GC issues)

As your application runs, the new objects are created in young generation. When the young generation is about to fill up, GC runs what is known as a ‘minor‘ collection and reclaims memory from unused objects in young generation. While at it, the minor collection also moves ‘long lived’ objects to ‘old’ generation. A ‘major‘ collection (a.k.a Full GC), which is invoked when there is serious unmet demand for memory, sweeps through the entire heap to reclaim memory.

Read More