Analyzing Java Heap problems Part 1: Basic actions and tools

You think that your shiny Java app has some memory issues but how do you find out if that is true and what is taking up all that memory? Knowing the potential problems is fine. Nevertheless you still have to find out your actual problems. There are several instruments available to help you analyse your Java application regarding its memory usage. I will tell you about increasing your maximum heap (most of you surely know  about that), looking at the memory of a running app, making heap dumps (on demand or on OutOfMemoryException) and analyzing the dumps.

Increasing maximum heap

The Java VM has a setting that defines the maximum amount of heap memory available to your application. It defaults to 64MB which is enough for many programs. If you have a larger application you should try to start it with that value increased by passing the -Xmx<size>m parameter to the VM at startup. <size> is the value in MBytes so just fiddle around with that. If your app is leaking memory that won’t help you for long so you have to find out *if* it leaks.

Looking at memory usage of a running application

You can use jconsole for a quick look at your applications resource usage. jconsole is part of the Sun JDK since Java 6. You can connect the jconsole to any running java applications on your computer or even reachable over network and offering the Java Management Extensions (JMX) over TCP. Non-leaking programs should have a memory graph like this:

You can see, that the memory fluctuates over time because of the garbage collection cycles. But overall it does not grow. Next we will look at an application that leaks memory:

Above we see that the garbage collector (GC) tries its best but the used memory is growing over time. If we see such behaviour we probably need a heap dump to analyze the issue further.

Making a heapdump

Basically you have two nice ways to get a heap dump of your application which you can look into at a later time:

  1. Use jmap (which is also part of the Sun JDK 6) to dump the heap of a running application to a file using a command line like jmap -dump:format=b,file=myheap.hprof <pid>
  2. Tell the VM to make a heap dump when an OutOfMemoryException occurs by adding -XX:+HeapDumpOnOutOfMemoryError to the VM parameters at startup. With another switch you can specify the path for the dumps: -XX:HeapDumpPath=jmxdata .

After you have obtained a dump of your application you certainly want to have a look at it and find the issues. You can start with Sun’s jhat which is also part of current JDKs. After supplying jhat the hprof-file you can point your browser to the integrated webserver of jhat and browse the heap looking for the objects that take up your memory.

That way you can get an idea of what objects lived in memory when the heap dump was made and how they were referenced.

Conclusion

We have seen many ways to perform memory diagnostics using only free tools which are part of the JDK from Sun. They are all nice but have their limitations. Especially jhat has problems with usability and performance when you examine larger heap dumps with it.

Next time I will show you how to use the Eclipse plugin MAT for analysis of heap dumps obtained in one of the above ways. So stay tuned!

Java solves all memory problems, or maybe not?

Many people think that Java’s Garbage Collector (GC) solves all of their memory management problems. It is true that the GC does a great job in many many real world situations. It really eases your life as software developer especially compared to programming in languages like C /C++ where memory management is a major PITA. Even there you can help yourself by using object systems with reference counting, smart pointers etc. but you have to be aware of this issue all the time.

So everything regarding memory is fine in Java?

Actually not really. Many Java developers do not think about code potentially leading to memory leaks. I would like to point out some problems we encountered. The problems can be divided into two categories:

  1. Native resources which have to be managed manually
  2. Listeners attached to central objects which are never removed again

Examples of native resources

Database connections, result sets and so on are a very common native resource that need manual management. JDBC is a real pain regarding resource management and especially Oracle is very susceptical to leaking those. Either you are very careful here or you use some framework to help you. If you do not want to go the whole way to a persistence framework like hibernate, iBatis or toplink a solution like Spring JDBCTemplate may help you a lot.

Another example is the JOGL TextRenderer which has to be manually disposed or you will leak texture memory  and soon run into resource problems.

Files/Streams and Sockets should be handled carefully too. In most cases you are more or less in the same boat with the C/C++ people but using finally can help you there.

Examples of listener leaks

Sometimes something innocent looking like a Swing Component can turn into a memory leak. We used JDateChooser one of our projects and found some of our data displaying dialogs to exist several times in memory and thus taking huge amounts of RAM eventually leading to OutOfMemoryExceptions. In case of dialogs and windows a WindowListener might help.

Sometimes you might write similar objects yourself that register to some central instance (maybe even a singleton *yuck*). Deregistering them always is easily forgotten or overlooked. A common code pattern to look out for listener leaks where you cannot deregister easily at the right moment is the following:

public class MyCoolClass implements IDataListener {

    public MyCoolClass(IDataProvider dataProvider) {
        super();
        dataProvider.addDataListener(this);
    }

    ...
}

Avoid such constructs as they can prove really dangerous. There is more that can be done to lower the risk of hard-hitting memory/listener leaks: Use WeakReferences for listener management at the crucial central objects. The referenced objects are taken care of by the GC and the listener manager has to take care of the WeakReferences. They can be cleaned up periodically or when a notification takes place.

Conclusion

The Java GC helps a lot in everyday programming but there are still things to look out for. Just be aware of the resources you are using and think about their need of management. I will write some follow up articles about getting heap dumps in different situations and searching them for memory leaks using some nice free tools.

Update:

Kris Kemper wrote a nice article about Swing Memory Leaks with JCalendar and a solution to the problem.

Deploying a Grails app on an Oracle DB

Running our new grails app on HSQL and a Postgresql everything went fine. But the production DB was decided to be an Oracle. And suddenly the app crashed several times. Here’s a list of what problems we encountered:

  • ORA-00972: identifier is too long
  • want to store a null value in a non null column

Oracle identifiers are limited to 30 characters. So we thought using a mapping for the table should do the trick. But grails uses the table names to construct the n:m relations and their id column names between the domain classes. Looking at the grails docs we found a joinTable mapping:

static mapping = {
    table 'PROP'
    tablePerHierarchy false
    instrumentInfos joinTable: [name:'PROP_INS', key:'id', column:'instrumentInfos_id']
}

This worked most of the time but in some cases grails just didn’t want to take our definitions. The problem was a bug in grails. The workaround we took was to shorten the domain classes names.
The second problem arose as we tried to store empty strings into the database. Oracle stores empty strings as null values which causes a constraint violation exception. The solution was to declare the string columns nullable or not nullable and not blank but you cannot use a not nullable and blank string with an Oracle DB.

Using Hudson for C++/CMake/CppUnit

Update: Hudson for C++/CMake/CppUnit Revised

As a follow-up to Using grails projects in Hudson, here is another not-so-standard usage of Hudson: C++ projects with CMake and CppUnit. Let’s see how that works out.

As long as you have Java/Ant/JUnit based projects, a fine tool that it is, configuration of Hudson is pretty straight forward. But if you have a C++ project with CMake as build system and CppUnit for your unit testing, you have to dig a little deeper. Fortunately, Hudson provides the possibility to execute arbitrary shell commands. So in order to build the project and execute the tests, we can simply put a shell script to work:

   # define build and installation directories
   BUILD_DIR=$WORKSPACE/build_dir
   INSTALL_DIR=$WORKSPACE/install_dir

   # we want to have a clean build
   rm -Rf $BUILD_DIR
   mkdir $BUILD_DIR
   cd $BUILD_DIR

   # initializing the build system
   cmake  ..  -DCMAKE_INSTALL_PREFIX=$INSTALL_DIR

   # fire-up the compiler
   make install

Environment variable WORKSPACE is defined by Hudson. Other useful variables are e.g. BUILD_NUMBER, BUILD_TAG and CVS_BRANCH.

But what about those unit tests? Hudson understands JUnit test result files out-of-the-box. So all we have to do is make CppUnit spit out an xml report and then translate it to JUnit form. To help us with that, we need a little xslt transformation. But first, let’s see how we can make CppUnit generate xml results (a little simplified):

#include <cppunit/necessary/CppUnitIncludes/>
...

using namespace std;
using namespace CppUnit;

int main(int argc, char** argv)
{
   TestResult    controller;
   TestResultCollector result;
   controller.addListener(&result);

   CppUnit::TextUi::TestRunner runner;
   runner.addTest( TestFactoryRegistry::getRegistry().makeTest() );
   runner.run(controller);

   // important stuff happens next
   ofstream xmlFileOut("cpptestresults.xml");
   XmlOutputter xmlOut(&result, xmlFileOut);
   xmlOut.write();
}

The assumption here is that your unit tests are built into libraries that are linked with the main function above. To execute the unit tests we add the following to out shell script:

   export PATH=$INSTALL_DIR/bin:$PATH
   export LD_LIBRARY_PATH=$INSTALL_DIR/lib:$LD_LIBRARY_PATH

   # call the cppunit executable
   cd $WORKSPACE
   cppunittests

This results in CppUnit generating file $WORKSPACE/cpptestresults.xml. Now, with the help of a little program called xsltproc and the following little piece of XSLT code, we can translate cpptestresults.xml to testresults.xml in JUnit format.

 <?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
    <xsl:output method="xml" indent="yes"/>
    <xsl:template match="/">
        <testsuite>
            <xsl:attribute name="errors"><xsl:value-of select="TestRun/Statistics/Errors"/></xsl:attribute>
            <xsl:attribute name="failures">
                <xsl:value-of select="TestRun/Statistics/Failures"/>
            </xsl:attribute>
            <xsl:attribute name="tests">
                <xsl:value-of select="TestRun/Statistics/Tests"/>
            </xsl:attribute>
            <xsl:attribute name="name">from cppunit</xsl:attribute>
            <xsl:apply-templates/>
        </testsuite>
    </xsl:template>
    <xsl:template match="/TestRun/SuccessfulTests/Test">
        <testcase>
            <xsl:attribute name="classname" ><xsl:value-of select="substring-before(Name, '::')"/></xsl:attribute>
            <xsl:attribute name="name"><xsl:value-of select="substring-after(Name, '::')"/></xsl:attribute>
        </testcase>
    </xsl:template>
    <xsl:template match="/TestRun/FailedTests/FailedTest">
        <testcase>
            <xsl:attribute name="classname" ><xsl:value-of select="substring-before(Name, '::')"/></xsl:attribute>
            <xsl:attribute name="name"><xsl:value-of select="substring-after(Name, '::')"/></xsl:attribute>
            <error>
                <xsl:attribute name="message">
                    <xsl:value-of select=" normalize-space(Message)"/>
                </xsl:attribute>
                <xsl:attribute name="type">
                    <xsl:value-of select="FailureType"/>
                </xsl:attribute>
                <xsl:value-of select="Message"/>
                File:<xsl:value-of select="Location/File"/>
                Line:<xsl:value-of select="Location/Line"/>
            </error>
        </testcase>
    </xsl:template>
    <xsl:template match="text()|@*"/>
</xsl:stylesheet>

The following call goes into our shell script:

xsltproc cppunit2junit.xsl $WORKSPACE/cpptestresults.xml > $WORKSPACE/testresults.xml

In the configuration page we can now check “Display JUnit test results” and give testresults.xml as result file. As a last step, we can package everything in $WORKSPACE/install_dir into a .tgz file and have Hudson to store it as build artifact. That’s it!

As always, there is room for improvements. One would be to wrap the shell script code above in a separate bash script and have Hudson simply call that script. The only advantage of the approach above is that you can see what’s going on directly on the configuration page. If your project is bigger, you might have more than one CppUnit executable. In this case, you can for example generate all testresult.xml files into a separate directory and tell Hudson to take into account all .xml files there.

Update: For the CMake related part of the above shell script I recently published the first version of a cmakebuilder plugin for Hudson. Check out my corresponding blog post.

Global error pages with Jetty and grails

We wanted to configure global error pages for our grails app. Using your favorite search engine you quickly find the following info.

At the bottom it says that in order to support global error pages you have to forward to a context because error pages can only be handled within contexts/webapps.
So I started adding a context to the contexts dir which looked like this:

<Configure class="org.mortbay.jetty.webapp.WebAppContext">
      <Set name="contextPath">/</Set>
      <Set name="war"><SystemProperty name="jetty.home" default="."/>/webapps/mywebapp.war</Set>
      <Set name="extractWAR">false</Set>
...

This caused an “IllegalArgumentException: name” at startup. The solution was to set extractWAR to true. JSPs or other resources (like GSPs) cannot be used inside a war when extractWAR is set to false. But this way got another pitfall: using localhost:8080/mywebapp won’t work. So why not just forward all requests from / to /mywebapp. Said and done:

<Set name="handler">
  <New id="Handlers" class="org.mortbay.jetty.handler.RewriteHandler">
    <Set name="rewriteRequestURI">false</Set>
    <Set name="rewritePathInfo">false</Set>
    <Set name="originalPathAttribute">requestedPath</Set>
    <Call name="addRewriteRule"><Arg>/mywebapp/*</Arg><Arg></Arg></Call>
    <Call name="addRewriteRule"><Arg>/*</Arg><Arg>/mywebapp</Arg></Call>
    <Set name="handler">
here the old handlers are inserted...

Now /mywebapp points to my webapp. / gives a 500 and other invalid urls give a 404.
To use your custom error pages inside a grails app just add the error codes you want to map inside the UrlMappings.groovy file:

class UrlMappings {
  ...
  static mappings = {
    "500"(view:'/error')
    "404"(view:'/error404')
  }
}

Using grails projects in Hudson

Being an agile software development company we use a continuous integration (CI) server like Hudson.
For our grails projects we wrote a simple ant target -call-grails to call the batch or the shell scripts:

    <condition property="grails" value="${grails.home}/bin/grails.bat">
        <os family="windows"/>
    </condition>
    <property name="grails" value="${grails.home}/bin/grails"/>

    <target name="-call-grails">
		<chmod file="${grails}" perm="u+x"/>
        <exec dir="${basedir}" executable="${grails}" failonerror="true">
            <arg value="${grails.task}"/>
            <arg value="${grails.file.path}"/>
            <env key="GRAILS_HOME" value="${grails.home}"/>
        </exec>
    </target>

Calling it is as easy as calling any ant target:

  <target name="war" description="--> Creates a WAR of a Grails application">
        <antcall target="-call-grails">
            <param name="grails.task" value="war"/>
            <param name="grails.file.path" value="${target.directory}/${artifact.name}"/>
        </antcall>
    </target>

One pitfall exists though, if your target takes no argument(s) after the task you have to use a different call:

	<target name="-call-grails-without-filepath">
		<chmod file="${grails}" perm="u+x"/>
        <exec dir="${basedir}" executable="${grails}" failonerror="true">
            <arg value="${grails.task}"/>
            <env key="GRAILS_HOME" value="${grails.home}"/>
        </exec>
    </target>

== or equals with Java enum

When you compare objects in Java you should prefer the equals()-method to == in general. The reason is that you get reference equality (like with ==) by default but you are able to change that behaviour. You can override equals() (DO NOT FORGET TO OVERRIDE hashCode() too because otherwise you will break the general class contract) to reflect logical equality which is often what you want, e.g when comparing some string constant with user input.

With primitive types like double and int you are more or less limited to == which is fine for those immutable value types.

But what is the right thing to to with the enum type introduced in Java 5?
Since enums look like a class with methods, fields and the like you might want to use equals() instead of ==. Now this is a special case where using reference equality is actually safer and thus better than logical equality.

Above (please mind the stupid example) we can see that comparing the EState enum with an ILamp using equals() is accepted perfectly by the compiler even though the condition never can be true in practice. Using == the compiler screams and tells us that we are comparing apples with oranges.

Using Flying Saucer PDF offline

Flying saucer is a nice tool for quick PDF generation from a (X)HTML page. Everything worked fine when we tested it at home but when we had a demo at a client’s site, no PDF could be generated. The problem was caused by a little snippet in the header of the HTML:

<!DOCTYPE html PUBLIC
"-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

The DTD declaration! So we took a look at the flying saucer issue database and found someone who had the same problem.
But another solution is even simpler:
Xerces has a default setting which tells the parser to load external dtds.
Turning this off solved our offline problems:

  DocumentBuilderFactory builderFactory =
    DocumentBuilderFactory.newInstance();
  builderFactory.setFeature(
    "http://apache.org/xml/features/nonvalidating/load-external-dtd",
    false);

When as Set is not what you want

When you want to filter out duplicates in a list in groovy you normally do something like:

        def list = [2, 1, 2, 3]
        def filtered = list as Set
        assertEquals([1, 2, 3], filtered as List)

This kicks out all duplicates in a one-liner. But what if the list is sorted (e.g. in reverseOrder)?

        def list = [3, 2, 2, 1]
        def filtered = list as Set
        assertEquals([3, 2, 1], filtered as List) // this fails!

One solution would be to use a small closure:

        def list = [3, 2, 2, 1]
        def filteredList = []
        list.each {
            if (!filteredList.contains(it)) {
                filteredList << it
            }
        }
        assertEquals([3, 2, 1], filteredList)

This closures preserves the order and filters out the duplicates.

Multipage Flows with Grails Part One – The traditional way

When you are developing web apps once in a while you need a flow of multiple pages just like a shopping cart at Amazon. Since there are different ways to implement such a multipage flow in Grails I thought of recording our experiences here.

The scenario

We model the process of submission of a proposal for different kinds of technologies. First you have to decide which type of proposal (once, long-term, in house) you want to submit. On the second page you give your personal infos and a general description of your intentions using the chosen technologies. Here you also choose the technologies you want. This has an impact on the following page which holds the information for the different technologies. Last but not least you save the proposal and show it to the user so he can see what data he entered. This flow could look like this:

-> chooseType -> enterGeneralInfo -> enterSpecificInfo -> save -> show

In each of the steps we need different parts of the full proposal. So we decided to use Command Objects in Grails. These objects are filled and are validated automatically by Grails. To use them you just have to delare them as parameter to the closure:

def enterSpecificInfo = {GeneralPartCommand cmd ->
  if (!cmd.hasErrors()) {
    Proposal proposal = new Proposal(cmd.properties)
    proposal.specificInfos = determineSpecificInfos(proposal.technologies)
    render(view: 'enterSpecificInfo', model: ['proposal': proposal])
  }
  else {
    render(view: 'enterGeneralInfo', model: ['proposal': cmd])
  }
}

You can then bind the properties in the command object to the proposal itself. The GeneralPartCommand has the properties and constraints which are needed for the general information of the proposal. This imposes a violation of the DRY (Don’t Repeat Yourself) principle as the properties and the constraints are duplicated in the proposal and the command object but more about this in another post.

How to collect all the data needed

Since you can only and possibly want to save the proposal at the end of the flow when it is complete you have to track the information entered in the previous steps. Here hidden form fields come in handy. These have to store the information of all previous steps taken so far. This can be a burden to do by hand and is a place where bugs arise.

How to go from one state to another

To transition from one state to the next you just validate the command object (or the proposal in the final state) to see if all data required is there. If you have no errors you render the view of the next state. When something is missing you just re-render the current view:

if (!cmd.hasErrors()) {
  Proposal proposal = new Proposal(cmd.properties)
  proposal.specificInfos = determineSpecificInfos(proposal.technologies)
  render(view: 'enterSpecificInfo', model: ['proposal': proposal])
}
else {
  render(view: 'enterGeneralInfo', model: ['proposal': cmd])
}

The errors section in the view takes the errors and prints them out. The forms in the views then use the next state for the action parameter.

Summary

An advantage of this solution is you can use it without any add-ons or plugins. Also the URLs are simple and clean. You can use the back button and jump to almost every page in the flow without doing any harm. Almost – the transition which saves the proposal causes the same proposal to be saved again. This duplicates the proposal because the only key is the internal id which is set in the save action.
Besides the DRY violation sketched above another problem arises from the fact that the logic of the states and the transitions are scattered between several controller actions and the action parameters in the forms. Also you have to remember to track every data in hidden form fields.
In a future post we take a look at how Spring (WebFlow) solves some of these problems and also introduces others.