chenhualeguan 2013-10-16
这几天在解决一个用mina开发的高并发通信过程中产生的一个bug。
模拟场景为:
通过定时触发启动线程模拟高并发短连接测试,测试的服务端有2个,一个是服务有起,一个没起,客户端和服务端均在同一服务器上。执行一段时间后linux主机上通过lsof命令查看,发现有递增的文件句柄,pipe和eventpoll。
抛出的异常如下:
2012-10-13 10:09:48 -org.apache.mina.core.service.SimpleIoProcessorPool.<init>(SimpleIoProcessorPool.java:197) Failed to create a new instance of org.apache.mina.transport.socket.nio.NioProcessor:null java.lang.reflect.InvocationTargetException at sun.reflect.GeneratedConstructorAccessor110.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.mina.core.service.SimpleIoProcessorPool.<init>(SimpleIoProcessorPool.java:180) at org.apache.mina.core.service.SimpleIoProcessorPool.<init>(SimpleIoProcessorPool.java:112) at org.apache.mina.core.polling.AbstractPollingIoConnector.<init>(AbstractPollingIoConnector.java:93) at org.apache.mina.transport.socket.nio.NioSocketConnector.<init>(NioSocketConnector.java:56) at com.develop.webplatform.funnel.client.JobClient.sendMessage(JobClient.java:39) at com.develop.webplatform.funnel.client.JobClient.sendJob(JobClient.java:126) at com.develop.webplatform.funnel.extend.JobExecRemotelyBySocket.execJobByTask(JobExecRemotelyBySocket.java:66) at com.develop.webplatform.funnel.JobManager.execJobByTask(JobManager.java:27) at com.develop.webplatform.quartz.job.TaskJob.executeInternal(TaskJob.java:38) at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:86) at org.quartz.core.JobRunShell.run(JobRunShell.java:223) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549) Caused by: org.apache.mina.core.RuntimeIoException: Failed to open a selector. at org.apache.mina.transport.socket.nio.NioProcessor.<init>(NioProcessor.java:61) ... 15 more Caused by: java.io.IOException: Too many open files at sun.nio.ch.IOUtil.initPipe(Native Method) at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49) at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18) at java.nio.channels.Selector.open(Selector.java:209) at org.apache.mina.transport.socket.nio.NioProcessor.<init>(NioProcessor.java:59) ... 15 more
原代码中,关于客户端连接的代码如下:
final NioSocketConnector connector = new NioSocketConnector(); final String[] result = new String[1]; connector.getFilterChain().addLast("codec", new ProtocolCodecFilter(new ObjectSerializationCodecFactory())); connector.setHandler(handler); //设置超时 connector.setConnectTimeoutMillis(defaultConnectTimeOut); ConnectFuture connectFuture = connector.connect(address); connectFuture.awaitUninterruptibly(); //同步,等待,直到连接完成 if (connectFuture.isDone()) { if (!connectFuture.isConnected()) { //若在指定时间内没连接成功,则抛出异常 logger.info("fail to connect " + logInfo); throw new Exception(); } }
经过分析,导致主机文件句柄泄露的原因为,客户端发起服务端连接时,会请求系统分配相关的文件句柄,在原代码中,仅仅判断是否连接成功,而未对连接失败进行资源释放,从而造成文件句柄泄露。当总的文件句柄数超过系统设置值(ulimit -n 查看同一个进程允许的最大文件句柄数),则抛出异常“java.io.IOException: Too many open files",导致无法创建新的连接,服务器挂掉。
更改后的代码如下:
final NioSocketConnector connector = new NioSocketConnector(); final String[] result = new String[1]; connector.getFilterChain().addLast("codec", new ProtocolCodecFilter(new ObjectSerializationCodecFactory())); connector.setHandler(handler); //设置超时 connector.setConnectTimeoutMillis(defaultConnectTimeOut); ConnectFuture connectFuture = connector.connect(address); connectFuture.awaitUninterruptibly(); //同步,等待,直到连接完成 if (connectFuture.isDone()) { if (!connectFuture.isConnected()) { //若在指定时间内没连接成功,则抛出异常 logger.info("fail to connect " + logInfo); connector.dispose(); //不关闭的话会运行一段时间后抛出,too many open files异常,导致无法连接 throw new Exception(); } }