Ever looked at the Windows task manager to see what CPU and memory usage the Tomcat process is using, but wanted to know more? The following describes how to configure Java monitoring and a free tool to monitor the web server resource usage, and also be able to identify which Dekho HTTP connection that may be consuming it.
Configuring Java Management Extensions (JMX)
- On the Tomcat web server, run the tomcat6w.exe configuration utility. This will by default be installed to C:\Program Files\Apache Software Foundation\Tomcat 6.0\bin
- On the Java tab, add the following switches to the Java Options
-Djava.rmi.server.hostnam=<insert server name>
- Security can be Optionally enabled, but can be very fiddly to get working. To get it working; add the following Java Option switches, add the access and password file, and remove all file permissions to access it, except to the user that the Tomcat Windows service runs under. Which means as a admin user you should Not have access to the folder after it is done correctly. This is to protect the clear text password.
- Restart the tomcat windows service
Conecting a Monitoring tool to JMX
- Download VisualVM. It will require the Java Developer Kit to run. http://visualvm.java.net/
- Install it on a remote machine so as not to place a small load on the production server. Installing on the Tomcat server will however enable extra memory profiling options in VisualVM.
- Add a connection to the Tomcat web server.
The monitoring tab in VisualVM shows the Java virtual machine CPU usage and memory usage.
The CPU usage gives us extra detail about how much garbage collection (GC) is impacting on the overall web server. The GC usage should be very small.
The Heap usage gives extra detail of what is actually used by the Java VM. Where as the Windows task manager only shows approximately the total allocated heap size plus PermGen heap size.
The Perform GC button, forces a major garbage collection to occur.
The heap dump button, generates a heap dump on the web server. This takes a few seconds. Note, that it also performs a major garbage collection before this occurs, so you will possibly notice a large amount of heap space free up after performing a garbage collection.
Java Heap Usage and Garbage Collection
The default Java Garbage Collector runs minor and major garbage collections. Looking at the allocated heap below, there are lots of minor collections, and one major garbage collection at 9:26am. One can also notice the CPU GC Activity at 9:26am. A major garbage collection will run when it is necessary to claim back some memory. Depending on available heap memory, it could be an hour or more between a major garbage collection. This minimises the CPU impact.
The Java heap is split into 3 types
- Young (Eden) Heap
New Java objects are created in this heap. If memory is required, a minor garbage collection is performed on the Eden heap. Objects surviving several minor garbage collections, get copied (not moved) to the Tenured heap.
The size of this Eden heap in recent Java implementations is by default set to ratio of 1:2 of the tenured heap. That is 1/3 of the total heap size. In the above heap usage diagram, the Eden heap would be approximately 500MB of the total 1536MB heap size. The minor collections are returning approximately 300MB.
- Tenured Heap
This contains older Java objects that have survived several garbage collections in the Eden heap. If tenured heap memory is required when copying objects from the Eden heap, a major garbage collection is performed on the tenured heap. Since this heap is larger, it takes longer to collect, using more CPU resources. The size of the tenured heap is by default is 2/3 of the total heap size. In the above heap usage diagram, the Tenured heap would be approximately 1000MB. The observed major collections above at 9:26am is returning approximately all of the 1000MB of the total 1536MB heap size.
- Permgen Heap
The permgen heap contains data describing the allocation of the Eden and Tenured heaps. That is, metadata of the allocated objects. The size of the permgen heap is set by the -XX:MaxPermGenSize setting.
Another example of minor and major garbage collections.
Java CPU Usage
To further diagnose CPU usage like the example below which is using a single CPU core (25%) for over 10 minutes, switch VisualVM to the Sampler tab, and click the CPU sampling button.
After about 30 seconds, click the Snapshot button above. This will save a snapshot of the CPU usage in this time period.
Open the snapshot and sort it by the Time (CPU) column. Look for the first http-80-xx thread and expand it. Ignore http-80-Poller threads which are waiting for new connections.
Also depending on the Dekho log level, you will be able to search (grep is easier) the tomcat log files for [http-80-xx] to find out what the user was doing.
FYI the cause of the following issue is thought to be the selection of a large number of features. Still being investigating the Dekho configuration.
We can see that http-80-24 thread is working hard performing a deflate() compression of some data. Taking subsequent snapshots we can see that this http thread in Tomcat is busy for quite a while performing this operation.
Tomcat thread http-80-24 is now writing the data to the HTTP stream to be sent to the Dekho client.