高峰期,程序卡顿 日志中找到堆栈dump, 发现大量请求在等待数据库连接。
"http-apr-7001-exec-146" Id=4693 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5727538a at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5727538a at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source) at com.alibaba.druid.pool.DruidDataSource.pollLast(DruidDataSource.java:1568) at com.alibaba.druid.pool.DruidDataSource.getConnectionInternal(DruidDataSource.java:1144) at com.alibaba.druid.pool.DruidDataSource.getConnectionDirect(DruidDataSource.java:1017)
数据库配置的100连接,看来用光了。
然后发现一些线程卡在归还connection上
"http-apr-7001-exec-249" Id=4797 WAITING on java.util.concurrent.locks.ReentrantLock$FairSync@43f6891 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.ReentrantLock$FairSync@43f6891 at java.util.concurrent.locks.LockSupport.park(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(Unknown Source) at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(Unknown Source) at com.alibaba.druid.pool.DruidDataSource.recycle(DruidDataSource.java:1338) at com.alibaba.druid.pool.DruidPooledConnection.recycle(DruidPooledConnection.java:305) at com.alibaba.druid.filter.FilterChainImpl.dataSource_recycle(FilterChainImpl.java:4534) at com.alibaba.druid.filter.stat.StatFilter.dataSource_releaseConnection(StatFilter.java:647) at com.alibaba.druid.filter.FilterChainImpl.dataSource_recycle(FilterChainImpl.java:4530) at com.alibaba.druid.pool.DruidPooledConnection.close(DruidPooledConnection.java:253) at org.springframework.jdbc.datasource.DataSourceUtils.doCloseConnection(DataSourceUtils.java:341) at org.springframework.jdbc.datasource.DataSourceUtils.doReleaseConnection(DataSourceUtils.java:328)
一些卡在获取connection上
"http-apr-7001-exec-204" Id=4752 WAITING on java.util.concurrent.locks.ReentrantLock$FairSync@43f6891 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.ReentrantLock$FairSync@43f6891 at java.util.concurrent.locks.LockSupport.park(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(Unknown Source) at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(Unknown Source) at com.alibaba.druid.pool.DruidDataSource.getConnectionInternal(DruidDataSource.java:1126) at com.alibaba.druid.pool.DruidDataSource.getConnectionDirect(DruidDataSource.java:1017) at com.alibaba.druid.filter.FilterChainImpl.dataSource_connect(FilterChainImpl.java:4544) at com.alibaba.druid.filter.stat.StatFilter.dataSource_getConnection(StatFilter.java:662) at com.alibaba.druid.filter.FilterChainImpl.dataSource_connect(FilterChainImpl.java:4540) at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:995)
注意,它们在竞争公平锁。java.util.concurrent.locks.ReentrantLock$FairSync@43f6891
负责创建连接的线程,也在等待公平锁。
"Druid-ConnectionPool-Create-297239085" Id=15 WAITING on java.util.concurrent.locks.ReentrantLock$FairSync@43f6891 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.ReentrantLock$FairSync@43f6891 at java.util.concurrent.locks.LockSupport.park(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Unknown Source) at java.util.concurrent.locks.ReentrantLock$FairSync.lock(Unknown Source) at java.util.concurrent.locks.ReentrantLock.lock(Unknown Source) at com.alibaba.druid.pool.DruidDataSource.put(DruidDataSource.java:1815) at com.alibaba.druid.pool.DruidDataSource$CreateConnectionThread.run(DruidDataSource.java:2033)
搜了下,貌似公平锁影响性能。 大概是这样: 比如5个线程去释放连接,100个请求去获取连接,后者明显多, 公平锁,大家机会平等?然后大概率前5个得不到锁,所以释放不掉连接。哈哈。
看了下druid的源代码:
public void setMaxWait(long maxWaitMillis) { if (maxWaitMillis != this.maxWait) { if (maxWaitMillis > 0L && this.useUnfairLock == null && !this.inited) { ReentrantLock lock = this.lock; lock.lock();
try { if (!this.inited && !lock.isFair()) { this.lock = new ReentrantLock(true); this.notEmpty = this.lock.newCondition(); this.empty = this.lock.newCondition(); } } finally { lock.unlock(); } }
if (this.inited) { LOG.error("maxWait changed : " + this.maxWait + " -> " + maxWaitMillis); }
this.maxWait = maxWaitMillis; } }
setMaxWait的时候,会使用公平锁(如果没有设置useUnfairLock属性)。
public void setUseUnfairLock(boolean useUnfairLock) { if (this.lock.isFair() != !useUnfairLock) { if (!this.inited) { ReentrantLock lock = this.lock; lock.lock();
try { if (!this.inited) { this.lock = new ReentrantLock(!useUnfairLock); this.notEmpty = this.lock.newCondition(); this.empty = this.lock.newCondition(); this.useUnfairLock = useUnfairLock; } } finally { lock.unlock(); } }
} }
setUseUnfairLock时会使用非公平锁,并设置this.useUnfairLock属性。 所以setUseUnfairLock和setMaxWait谁先执行,无所谓,都会变成非公平锁。
我的程序里,只设置了maxWait,没写UseUnFaireLock,所以是公平锁。 那么增加UseUnFaireLock为true即可。
当然,最大连接数,还是需要改大一些。
参考文章: https://tech.youzan.com/you-zan-shu-ju-ku-lian-jie-chi-xing-neng-you-hua/ https://blog.csdn.net/beFocused/article/details/108533137
|