某客户系统用着用着就挂了,看了半天日志,发现是JVM崩溃了。
2021-08-28T10:00:01.134+0800: [GC [PSYoungGen: 2833245K->11997K(2845184K)] 4091506K->1273237K(4123136K), 0.0188348 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 2021-08-28T10:21:49.270+0800: [GC [PSYoungGen: 2835165K->12998K(2845696K)] 4096405K->1278792K(4123648K), 0.0209580 secs] [Times: user=0.08 sys=0.00, real=0.02 secs] 2021-08-28T10:21:49.312+0800: [Full GC# # A fatal error has been detected by the Java Runtime Environment: # # EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x000000005ea105e8, pid=4556, tid=3548 # # JRE version: Java(TM) SE Runtime Environment (7.0_79-b15) (build 1.7.0_79-b15) # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.79-b02 mixed mode windows-amd64 compressed oops) # Problematic frame: # V [jvm.dll+0xf05e8]
# # A fatal error has been detected by the Java Runtime Environment: # # EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x000000005ea105e8, pid=4556, tid=3548 # # JRE version: Java(TM) SE Runtime Environment (7.0_79-b15) (build 1.7.0_79-b15) # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.79-b02 mixed mode windows-amd64 compressed oops) # Problematic frame: # V [jvm.dll+0xf05e8] # # Failed to write core dump. Minidumps are not enabled by default on client versions of Windows # # If you would like to submit a bug report, please visit: # http://bugreport.java.com/bugreport/crash.jsp #
--------------- T H R E A D ---------------
Current thread (0x00000000014aa000): GCTaskThread [stack: 0x00000000053e0000,0x00000000054e0000] [id=3548]
siginfo: ExceptionCode=0xc0000005, reading address 0x0000000003080378
看了heap信息,确实是老年代满了,触发FullGC时,因为什么原因,JVM崩溃了。
(1)搜到这个,说JVM的GC对内存好坏极其敏感,因为内存里有大量的地址指向,有一个出错可能就影响极大。 而JVM将内存读写错误包装了一层,所以看起来像JVM的问题,实际是内存的问题。 https://shipilev.net/jvm/test-your-memory/ 找了个memtest软件,随便扫了下,就发现100多个错误。 看来,换内存条是首选,但是客户不同意。
(2)可以换一种GC方式试一试,或者升级JDK版本。由于不知道是否兼容,不敢随便升级JDK,那是自找麻烦。 对于内存回收的错误,一般采取改变回收的算法和参数的方法来绕过去。
(3)老年代分了1G多,发现几百M时,用jmap -histo:live 1111触发fullGC,是没问题的。 实测老年代400多M就够用,所以干脆就留了600M,其他都给年轻代。 跑了一天,fullGC了两次,没有问题。 先凑合着……
|