lankk的魔法书札 2019-09-08
keys *
这个命令千万别在生产环境乱用。特别是数据庞大的情况下。因为Keys会引发Redis锁,并且增加Redis的CPU占用。很多公司的运维都是禁止了这个命令的
当需要扫描key,匹配出自己需要的key时,可以使用 scan
命令
scan
操作的Helper实现import java.io.IOException; import java.nio.charset.StandardCharsets; import java.util.ArrayList; import java.util.List; import java.util.function.Consumer; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.data.redis.connection.RedisConnection; import org.springframework.data.redis.core.Cursor; import org.springframework.data.redis.core.ScanOptions; import org.springframework.data.redis.core.StringRedisTemplate; import org.springframework.stereotype.Component; @Component public class RedisHelper { @Autowired private StringRedisTemplate stringRedisTemplate; /** * scan 实现 * @param pattern 表达式 * @param consumer 对迭代到的key进行操作 */ public void scan(String pattern, Consumer<byte[]> consumer) { this.stringRedisTemplate.execute((RedisConnection connection) -> { try (Cursor<byte[]> cursor = connection.scan(ScanOptions.scanOptions().count(Long.MAX_VALUE).match(pattern).build())) { cursor.forEachRemaining(consumer); return null; } catch (IOException e) { e.printStackTrace(); throw new RuntimeException(e); } }); } /** * 获取符合条件的key * @param pattern 表达式 * @return */ public List<String> keys(String pattern) { List<String> keys = new ArrayList<>(); this.scan(pattern, item -> { //符合条件的key String key = new String(item,StandardCharsets.UTF_8); keys.add(key); }); return keys; } }
但是会有一个问题:没法移动cursor,也只能scan一次,并且容易导致redis链接报错
http://doc.redisfans.com/key/scan.html
.count(Integer.MAX_VALUE)
,一下子全查回来;但是这样子和 keys 有啥区别呢?搞笑脸 & 疑问脸(JedisCommands) connection.getNativeConnection()
的 hscan、sscan、zscan 方法实现cursor遍历,参照下文2.2章节try { Cursor<Map.Entry<Object,Object>> cursor = redisTemplate.opsForHash().scan("field", ScanOptions.scanOptions().match("*").count(1000).build()); while (cursor.hasNext()) { Object key = cursor.next().getKey(); Object valueSet = cursor.next().getValue(); } //关闭cursor cursor.close(); } catch (IOException e) { e.printStackTrace(); }
client lists
`info clients`info stats
命令查看客户端连接状态,会发现scan操作一直存在client list ...... id=1531156 addr=xxx:55845 fd=8 name= age=80 idle=11 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=scan ...... org.springframework.data.redis.core.RedisTemplate#execute(org.springframework.data.redis.core.RedisCallback<T>, boolean, boolean) finally { RedisConnectionUtils.releaseConnection(conn, factory); }
public Set<String> scan(String matchKey) { Set<String> keys = redisTemplate.execute((RedisCallback<Set<String>>) connection -> { Set<String> keysTmp = new HashSet<>(); Cursor<byte[]> cursor = connection.scan(new ScanOptions.ScanOptionsBuilder().match("*" + matchKey + "*").count(1000).build()); while (cursor.hasNext()) { keysTmp.add(new String(cursor.next())); } return keysTmp; }); return keys; }
connection.getNativeConnection
;connection.getNativeConnection()
实际对象是Jedis(debug可以看出) ,Jedis实现了很多接口public class Jedis extends BinaryJedis implements JedisCommands, MultiKeyCommands, AdvancedJedisCommands, ScriptingCommands, BasicCommands, ClusterCommands, SentinelCommands
public Set<String> scan(String key) { return redisTemplate.execute((RedisCallback<Set<String>>) connection -> { Set<String> keys = Sets.newHashSet(); JedisCommands commands = (JedisCommands) connection.getNativeConnection(); MultiKeyCommands multiKeyCommands = (MultiKeyCommands) commands; ScanParams scanParams = new ScanParams(); scanParams.match("*" + key + "*"); scanParams.count(1000); ScanResult<String> scan = multiKeyCommands.scan("0", scanParams); while (null != scan.getStringCursor()) { keys.addAll(scan.getResult()); if (!StringUtils.equals("0", scan.getStringCursor())) { scan = multiKeyCommands.scan(scan.getStringCursor(), scanParams); continue; } else { break; } } return keys; }); }
client lists
`info clients`info stats
查看config get maxclients
查询redis允许的最大连接数 是 100001) "maxclients" 2) "10000"`
redis-cli
在其他机器上也可以直接登录 操作综上,redis本身没有卡死
netstat
查看和redis的连接,6333是redis端口;连接一直存在➜ ~ netstat -an | grep 6333 netstat -an | grep 6333 tcp4 0 0 xx.xx.xx.aa.52981 xx.xx.xx.bb.6333 ESTABLISHED tcp4 0 0 xx.xx.xx.aa.52979 xx.xx.xx.bb.6333 ESTABLISHED tcp4 0 0 xx.xx.xx.aa.52976 xx.xx.xx.bb.6333 ESTABLISHED tcp4 0 0 xx.xx.xx.aa.52971 xx.xx.xx.bb.6333 ESTABLISHED tcp4 0 0 xx.xx.xx.aa.52969 xx.xx.xx.bb.6333 ESTABLISHED tcp4 0 0 xx.xx.xx.aa.52967 xx.xx.xx.bb.6333 ESTABLISHED tcp4 0 0 xx.xx.xx.aa.52964 xx.xx.xx.bb.6333 ESTABLISHED tcp4 0 0 xx.xx.xx.aa.52961 xx.xx.xx.bb.6333 ESTABLISHED
jstack
查看应用的堆栈信息"http-nio-7007-exec-2" #139 daemon prio=5 os_prio=31 tid=0x00007fda36c1c000 nid=0xdd03 waiting on condition [0x00007000171ff000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000006c26ef560> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at org.apache.commons.pool2.impl.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:590) at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:441) at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:362) at redis.clients.util.Pool.getResource(Pool.java:49) at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226) at redis.clients.jedis.JedisPool.getResource(JedisPool.java:16) at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:276) at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:469) at org.springframework.data.redis.core.RedisConnectionUtils.doGetConnection(RedisConnectionUtils.java:132) at org.springframework.data.redis.core.RedisTemplate.executeWithStickyConnection(RedisTemplate.java:371) at org.springframework.data.redis.core.DefaultHashOperations.scan(DefaultHashOperations.java:244)
综上,是应用侧卡死
client lists
显示 scan 连接还在,没有释放;应用线程也还是处于卡死状态config get timeout
,redis未设置超时时间,可以用 config set timeout xxx
设置,单位秒;但是设置了redis的超时,redis释放了连接,应用还是一样卡住1) "timeout" 2) "0"
netstat
查看和redis的连接,6333是redis端口;连接从ESTABLISHED变成了CLOSE_WAIT;jstack
和 原来表现一样,卡在JedisConnectionFactory.getConnection
➜ ~ netstat -an | grep 6333 netstat -an | grep 6333 tcp4 0 0 xx.xx.xx.aa.52981 xx.xx.xx.bb.6333 CLOSE_WAIT tcp4 0 0 xx.xx.xx.aa.52979 xx.xx.xx.bb.6333 CLOSE_WAIT tcp4 0 0 xx.xx.xx.aa.52976 xx.xx.xx.bb.6333 CLOSE_WAIT tcp4 0 0 xx.xx.xx.aa.52971 xx.xx.xx.bb.6333 CLOSE_WAIT tcp4 0 0 xx.xx.xx.aa.52969 xx.xx.xx.bb.6333 CLOSE_WAIT tcp4 0 0 xx.xx.xx.aa.52967 xx.xx.xx.bb.6333 CLOSE_WAIT tcp4 0 0 xx.xx.xx.aa.52964 xx.xx.xx.bb.6333 CLOSE_WAIT tcp4 0 0 xx.xx.xx.aa.52961 xx.xx.xx.bb.6333 CLOSE_WAIT
netstat -an
基本可以确定 redis 连接池的大小是 8 ;结合代码配置,没有指定的话,默认也确实是8redis.clients.jedis.JedisPoolConfig private int maxTotal = 8; private int maxIdle = 8; private int minIdle = 0;
@Bean public RedisConnectionFactory redisConnectionFactory() { RedisStandaloneConfiguration redisStandaloneConfiguration = new RedisStandaloneConfiguration(); redisStandaloneConfiguration.setHostName(redisHost); redisStandaloneConfiguration.setPort(redisPort); redisStandaloneConfiguration.setPassword(RedisPassword.of(redisPasswd)); JedisConnectionFactory cf = new JedisConnectionFactory(redisStandaloneConfiguration); cf.afterPropertiesSet(); return cf; }
readTimeout,connectTimeout不指定,有默认值 2000 ms
org.springframework.data.redis.connection.jedis.JedisConnectionFactory.MutableJedisClientConfiguration private Duration readTimeout = Duration.ofMillis(Protocol.DEFAULT_TIMEOUT); private Duration connectTimeout = Duration.ofMillis(Protocol.DEFAULT_TIMEOUT);
B. 修改后配置
@Bean public RedisConnectionFactory redisConnectionFactory() { JedisPoolConfig jedisPoolConfig = new JedisPoolConfig(); jedisPoolConfig.setMaxTotal(16); // --最多可以建立16个连接了 jedisPoolConfig.setMaxWaitMillis(10000); // --10s获取不到连接池的连接, // --直接报错Could not get a resource from the pool jedisPoolConfig.setMaxIdle(16); jedisPoolConfig.setMinIdle(0); JedisConnectionFactory cf = new JedisConnectionFactory(jedisPoolConfig); cf.setHostName(redisHost); // -- @Deprecated cf.setPort(redisPort); // -- @Deprecated cf.setPassword(redisPasswd); // -- @Deprecated cf.setTimeout(30000); // -- @Deprecated 貌似没生效,30s超时,没有关闭连接池的连接; // --redis没有设置超时,会一直ESTABLISHED;redis设置了超时,且超时之后,会一直CLOSE_WAIT cf.afterPropertiesSet(); return cf; }
RedisStandaloneConfiguration redisStandaloneConfiguration = new RedisStandaloneConfiguration(); redisStandaloneConfiguration.setHostName(redisHost); redisStandaloneConfiguration.setPort(redisPort); redisStandaloneConfiguration.setPassword(RedisPassword.of(redisPasswd)); JedisPoolConfig jedisPoolConfig = new JedisPoolConfig(); jedisPoolConfig.setMaxTotal(16); jedisPoolConfig.setMaxWaitMillis(10000); jedisPoolConfig.setMaxIdle(16); jedisPoolConfig.setMinIdle(0); cf = new JedisConnectionFactory(redisStandaloneConfiguration, JedisClientConfiguration.builder() .readTimeout(Duration.ofSeconds(30)) .connectTimeout(Duration.ofSeconds(30)) .usePooling().poolConfig(jedisPoolConfig).build());
redistemplate-游标scan使用注意事项