Home » Bugs, Debian, Java, Kernel, Linux

sun.jvm.hotspot.debugger.NoSuchSymbolException: Could not find symbol “gHotSpotVMTypeEntryTypeNameOffset”

8 February 2011 2 Comments

This, probably, is a bug which was not spotted very often because is very obscure and the Debian guys moved very quick and fixed the problem. I know, I found a lot of bugs related to sun.jvm.hotspot.debugger.NoSuchSymbolException: Could not find symbol “gHotSpotVMTypeEntryTypeNameOffset”, but this is different. Usually was a problem of “striping symbols from libjvm.so”, but in my case wasn’t that. Also I found this error related to OpenJDK (I use Sun JDK) and the solution was to use -server flag to get the correct libjvm.so. Anyway it took me one full day to discover what problem was.

In my case, this problem appear, in the following configuration:

Debian 5.0.x
Kernel 2.6.26-21lenny3
Sun Java 1.6.0_20

Running any from java “profiling” commands: jmap, jstack I get the following error:

$ jmap 2782
Attaching to process ID 2782, please wait...
sun.jvm.hotspot.debugger.NoSuchSymbolException: Could not find symbol "gHotSpotVMTypeEntryTypeNameOffset" in any of the known library names (libjvm.so, libjvm_g.so, gamma_g)
        at sun.jvm.hotspot.HotSpotTypeDataBase.lookupInProcess(HotSpotTypeDataBase.java:388)
        at sun.jvm.hotspot.HotSpotTypeDataBase.getLongValueFromProcess(HotSpotTypeDataBase.java:369)
        at sun.jvm.hotspot.HotSpotTypeDataBase.readVMTypes(HotSpotTypeDataBase.java:102)
        at sun.jvm.hotspot.HotSpotTypeDataBase.(HotSpotTypeDataBase.java:85)
        at sun.jvm.hotspot.bugspot.BugSpotAgent.setupVM(BugSpotAgent.java:568)
        at sun.jvm.hotspot.bugspot.BugSpotAgent.go(BugSpotAgent.java:494)
        at sun.jvm.hotspot.bugspot.BugSpotAgent.attach(BugSpotAgent.java:332)
        at sun.jvm.hotspot.tools.Tool.start(Tool.java:163)
        at sun.jvm.hotspot.tools.PMap.main(PMap.java:67)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at sun.tools.jmap.JMap.runTool(JMap.java:179)
        at sun.tools.jmap.JMap.main(JMap.java:110)
Debugger attached successfully.

If I looked on the symbols of libjvm.so, gHotSpotVMTypeEntryTypeNameOffset was there.

# nm /usr/lib/jvm/java-6-sun-1.6.0.20/jre/lib/amd64/server/libjvm.so | grep gHotSpotVMTypeEntryTypeNameOffset
0000000000a64ed8 b gHotSpotVMTypeEntryTypeNameOffset

So the problem was in other place.

I tried to see how the java is trying to find this symbol and I get the sources of HotSpotTypeDataBase. There if found the function with problems symbolLookup.lookup. But lookup is a wrapper for native “C” function Java_sun_jvm_hotspot_debugger_linux_LinuxDebuggerLocal_lookupByName0 which is another wrapper for lookup_symbol 🙂

As you see in the error and in the source code, the main parameters for all “lookup” functions is symbol and library name, but guess what I found in Java source code:

// ignore object_name. search in all libraries
// FIXME: what should we do with object_name?? The library names are obtained
// by parsing /proc//maps, which may not be the same as object_name.
// What we need is a utility to map object_name to real file name, something
// dlopen() does by looking at LD_LIBRARY_PATH and /etc/ld.so.cache. For
// now, we just ignore object_name and do a global search for the symbol.

So, truly, they don’t try to find this library and search this symbol in this files, just trying to get all open libraries opened by application and see if there is this symbol.

The next step was to see what libraries are checked by application and I used ltrace and LD_DEBUG=libs to do that, but I didn’t find nothing.

Seeing that, I tried to write my own application, to search this symbol, based on test.c which come with the hotspot sources.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
#include <stdio.h>
#include <stdlib.h>
#include "libproc.h"
 
int main(int argc, char** argv) {
   struct ps_prochandle* ph;
//initializing libproc
   init_libproc(true);
// grab handler from our running java application
   ph = Pgrab(atoi(argv[1]));
// try to search the symbol
    lookup_symbol(ph,argv[2],argv[3]);
// release handler
   if (ph) {
      Prelease(ph);
      return 0;
   }
}gHotSpotVMTypeEntryTypeNameOffset

where:
argv[1] is the pid of the application
argv[2] is the library name
argv[3] is the searched symbol

In a normal system you should get


./test2 5546 libjvm.so gHotSpotVMTypeEntryTypeNameOffset
libsaproc DEBUG: can't open shared object [heap]
libsaproc DEBUG: can't open shared object [stack]
libsaproc DEBUG: can't open shared object [vdso]
libsaproc DEBUG: can't open shared object [vsyscall]
libsaproc DEBUG: thread_db : pthread -1245444352 (lwp 5546)
libsaproc DEBUG: thread_db : pthread 2050656000 (lwp 5649)
libsaproc DEBUG: thread_db : pthread 2051708672 (lwp 5629)
libsaproc DEBUG: thread_db : pthread 2052761344 (lwp 5628)
libsaproc DEBUG: thread_db : pthread 2053814016 (lwp 5626)
[... more ...]

and in a buggy system you should get


# ./test2 2658 libjvm.so gHotSpotVMTypeEntryTypeNameOffset
libsaproc DEBUG: lookup failed for symbol 'nptl_version' in obj 'libpthread.so.0'
libsaproc DEBUG: can't create libthread_db agent
libsaproc DEBUG: lookup failed for symbol 'gHotSpotVMTypeEntryTypeNameOffset' in obj 'libjvm.so'

So the main problem was not with gHotSpotVMTypeEntryTypeNameOffset and with nptl_version from libpthread.so.0.

So the main problem come from libc/kernel. Just upgrading to the last version of kernel (2.6.26-26lenny2) and rebooting, solved my problem.

Warning:

Until 1.6.0_20 Debian packages stripped the symbols from java libraries. So first you should check this:

$ nm /usr/lib/jvm/java-6-sun-1.6.0.20/jre/lib/amd64/server/libjvm.so | grep gHotSpotVMTypeEntryTypeNameOffset

Update:
To compile my program, try to do a make in hotspot/agent/src/os/linux (which should fail, but it will build objects for the next step) and after run the following command:

gcc test2.c ps_proc.o ps_core.o libproc_impl.o salibelf.o symtab.o -lthread_db -o test2


2 Comments »

  • Bob Homes said:

    I was getting this same stupid error, but this here blog post helped me out of it. Thanks!

  • joe said:

    I have exactly the same issue, thank you for the post.

Leave your response!

Add your comment below, or trackback from your own site. You can also subscribe to these comments via RSS.

Be nice. Keep it clean. Stay on topic. No spam.

You can use these tags:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

This is a Gravatar-enabled weblog. To get your own globally-recognized-avatar, please register at Gravatar.