The HotSpot JVM was running Tomcat, which hosted our web application
in the production environment.
It had been restarted 6 days ago and had been running fine since. And
the same JVM+application usually work fine, we have used them for
months, restarting from time to time, and never had this problem.
In the JVM error file, I notice about 5000 threads, could this be a
problem ? Is the HotSpot JVM unable to handle many threads ? Should I
change the way my application uses threads ?
Could it be caused by my web application ?
I could not find any information on how to prevent this kind of
crashes.
What can I do so that it does not happen again ?
Below is the generated hs_err_pid*.log error file, in which I replaced
two long boring parts with "..."
Any idea/hint/advice appreciated ! :-)
Nicolas RAOUL.
-----------------------------------------------------------------
#
# An unexpected error has been detected by Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x064c5d46, pid=30568, tid=1797516208
#
# Java VM: Java HotSpot(TM) Server VM (1.6.0_01-b06 mixed mode)
# Problematic frame:
# V [libjvm.so+0x4c5d46]
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
#
--------------- T H R E A D ---------------
Current thread (0x080d0000): JavaThread "Timer-133" daemon
[_thread_blocked_trans, id=31497]
siginfo:si_signo=11, si_errno=0, si_code=0, si_addr=0x00000000
Registers:
EAX=0xb7f94000, EBX=0x080d0000, ECX=0x00000000, EDX=0x00000000
ESP=0x6b23ddc8, EBP=0x6b23de60, ESI=0x00000ffc, EDI=0x090180a4
EIP=0x064c5d46, CR2=0xb7f94000, EFLAGS=0x00010246
Top of Stack: (sp=0x6b23ddc8)
0x6b23ddc8: 0000000d 6b23dcd8 fffffffd 00000000
0x6b23ddd8: 090180f4 01000006 6b23de70 061876f0
0x6b23dde8: 080d0000 00000000 75b88818 75b88818
0x6b23ddf8: 080d0e20 00000002 6b23de30 062ddaba
0x6b23de08: 00000000 00000000 fffffff0 0023de30
0x6b23de18: 080d0000 34000010 0000002a b7f676e8
0x6b23de28: 6b23de28 6b23de28 080d0000 080d0d30
0x6b23de38: 00000000 00000003 00000000 00000029
Instructions: (pc=0x064c5d46)
0x064c5d36: 8b 35 b0 af 5a 06 c1 e9 03 a1 ac af 5a 06 21 f1
0x064c5d46: c7 04 01 01 00 00 00 e9 be fc ff ff 83 ec 0c 8b
Stack: [0x6b1ee000,0x6b23f000), sp=0x6b23ddc8, free space=319k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code,
C=native code)
V [libjvm.so+0x4c5d46]
V [libjvm.so+0x4c35ef]
V [libjvm.so+0x30785a]
J java.lang.Object.wait(J)V
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J java.lang.Object.wait(J)V
J java.util.TimerThread.mainLoop()V
j java.util.TimerThread.run()V+1
v ~StubRoutines::call_stub
--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
0x2a37ec00 JavaThread "Timer-4500" daemon [_thread_blocked_trans,
id=24820]
.... (4563 similar lines with "Timer-XYZ")
0x703ac800 JavaThread "Timer-3" daemon [_thread_blocked_trans,
id=30640]
0x7038c800 JavaThread "TP-Monitor" daemon [_thread_blocked,
id=30616]
0x6fce0c00 JavaThread "TP-Processor4" daemon [_thread_in_native,
id=30615]
0x70397c00 JavaThread "TP-Processor3" daemon [_thread_blocked,
id=30614]
0x70397800 JavaThread "TP-Processor2" daemon [_thread_blocked,
id=30613]
0x703c0c00 JavaThread "TP-Processor1" daemon [_thread_blocked,
id=30612]
0x703c5000 JavaThread "http-8080-Monitor" [_thread_blocked,
id=30611]
0x6fffe800 JavaThread
"ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon
[_thread_blocked_trans, id=30585]
0x6ffa1000 JavaThread "Thread-2" [_thread_blocked_trans, id=30582]
0x703ae400 JavaThread "Timer-0" daemon [_thread_blocked_trans,
id=30581]
0x08133400 JavaThread "Low Memory Detector" daemon [_thread_blocked,
id=30579]
0x08131800 JavaThread "CompilerThread1" daemon [_thread_blocked,
id=30578]
0x08130000 JavaThread "CompilerThread0" daemon [_thread_blocked,
id=30577]
0x0812ec00 JavaThread "Signal Dispatcher" daemon [_thread_blocked,
id=30576]
0x0811d400 JavaThread "Finalizer" daemon [_thread_blocked, id=30575]
0x0811cc00 JavaThread "Reference Handler" daemon [_thread_blocked,
id=30574]
0x08058800 JavaThread "main" [_thread_in_native, id=30570]
Other Threads:
0x0811a000 VMThread [id=30573]
0x08134c00 WatcherThread [id=30580]
VM state:synchronizing (normal execution)
VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
[0x08056aa0/0x08056ac8] Safepoint_lock - owner thread: 0x0811a000
[0x08056b20/0x08056b48] Threads_lock - owner thread: 0x0811a000
[0x08056fe0/0x08056ff8] Heap_lock - owner thread: 0x087a8400
Heap
PSYoungGen total 116288K, used 116272K [0xadcb0000, 0xb4e70000,
0xb4e70000)
eden space 116096K, 100% used [0xadcb0000,0xb4e10000,0xb4e10000)
from space 192K, 91% used [0xb4e10000,0xb4e3c060,0xb4e40000)
to space 192K, 0% used [0xb4e40000,0xb4e40000,0xb4e70000)
PSOldGen total 84416K, used 67409K [0x74e70000, 0x7a0e0000,
0xadcb0000)
object space 84416K, 79% used [0x74e70000,0x790444f0,0x7a0e0000)
PSPermGen total 18816K, used 18597K [0x70e70000, 0x720d0000,
0x74e70000)
object space 18816K, 98% used [0x70e70000,0x72099750,0x720d0000)
Dynamic libraries:
003bb000-003d0000 r-xp 00000000 fd:00 556129 /lib/ld-2.3.4.so
.... (9471 similar lines)
ffffe000-fffff000 ---p 00000000 00:00 0
VM Arguments:
jvm_args: -DTACTDIR=/usr/local/tact -DTACTDATADIR=/usr/local/
tactdatadir -DTACTURL=http://172.16.12.142:8080/tact
-Xms256m -
Xmx1024m -
Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -
Djava.util.logging.config.file=/usr/local/tomcat/conf/
logging.properties -Djava.endorsed.dirs=/usr/local/tomcat/common/
endorsed -Dcatalina.base=/usr/local/tomcat -Dcatalina.home=/usr/local/
tomcat -Djava.io.tmpdir=/usr/local/tomcat/temp
java_command: org.apache.catalina.startup.Bootstrap start
Launcher Type: SUN_STANDARD
Environment Variables:
JAVA_HOME=/usr/local/java
PATH=/usr/local/tactshells:/opt/firefox:/usr/local/java/bin:/usr/
kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/
bin:/usr/sbin:/usr/bin:/usr/local/tools:/usr/local/tactshells:/usr/
X11R6/bin:/home/tact/bin:.
LD_LIBRARY_PATH=/usr/local/jre1.6.0_01/lib/i386/server:/usr/local/
jre1.6.0_01/lib/i386:/usr/local/jre1.6.0_01/../lib/i386
SHELL=/bin/bash
DISPLAY=:0.0
Signal Handlers:
SIGSEGV: [libjvm.so+0x51d3a0], sa_mask[0]=0x7ffbfeff,
sa_flags=0x10000004
SIGBUS: [libjvm.so+0x51d3a0], sa_mask[0]=0x7ffbfeff,
sa_flags=0x10000004
SIGFPE: [libjvm.so+0x43d430], sa_mask[0]=0x7ffbfeff,
sa_flags=0x10000004
SIGPIPE: [libjvm.so+0x43d430], sa_mask[0]=0x7ffbfeff,
sa_flags=0x10000004
SIGILL: [libjvm.so+0x43d430], sa_mask[0]=0x7ffbfeff,
sa_flags=0x10000004
SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGUSR2: [libjvm.so+0x43f440], sa_mask[0]=0x00000000,
sa_flags=0x10000004
SIGHUP: [libjvm.so+0x43ee60], sa_mask[0]=0x7ffbfeff,
sa_flags=0x10000004
SIGINT: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGQUIT: [libjvm.so+0x43ee60], sa_mask[0]=0x7ffbfeff,
sa_flags=0x10000004
SIGTERM: [libjvm.so+0x43ee60], sa_mask[0]=0x7ffbfeff,
sa_flags=0x10000004
SIGUSR2: [libjvm.so+0x43f440], sa_mask[0]=0x00000000,
sa_flags=0x10000004
--------------- S Y S T E M ---------------
OS:Red Hat Enterprise Linux WS release 4 (Nahant Update 4)
uname:Linux 2.6.9-42.ELsmp #1 SMP Wed Jul 12 23:27:17 EDT 2006 i686
libc:glibc 2.3.4 NPTL 2.3.4
rlimit: STACK 10240k, CORE 0k, NPROC 32757, NOFILE 1024, AS infinity
load average:699.24 155.09 53.29
CPU:total 2 family 6, cmov, cx8, fxsr, mmx, sse, sse2
Memory: 4k page, physical 2074272k(473144k free), swap
2097144k(2097016k free)
vm_info: Java HotSpot(TM) Server VM (1.6.0_01-b06) for linux-x86,
built on Mar 14 2007 00:47:53 by "java_re" with gcc 3.2.1-7a (J2SE
release)
|
