我一直都有一個疑問,豐巢業務服務的生產環境jvm參數設置是禁止system.gc的,也就是開啟設置:-XX:+DisableExplicitGC,但是生產環境卻從來沒有出現過堆外記憶體溢出的情況。說明一下,豐巢使用了阿裡開源的dubbo,而dubbo底層通信預設情況下使用了3.2.5.Final版本的 ...
我一直都有一個疑問,豐巢業務服務的生產環境jvm參數設置是禁止system.gc的,也就是開啟設置:-XX:+DisableExplicitGC,但是生產環境卻從來沒有出現過堆外記憶體溢出的情況。說明一下,豐巢使用了阿裡開源的dubbo,而dubbo底層通信預設情況下使用了3.2.5.Final版本的netty,而我們對於netty的常規認知里,netty一定是使用了堆外記憶體,並且堆外記憶體在禁止了system.gc這個函數調用的話,在服務沒有主動回收分配的堆外記憶體的情況下,一定會出現堆外記憶體的泄露。帶著這個問題,剛好前天晚上有些時間,研究了一下3.2.5版本的netty源碼,又是在科興科興園等饅頭媽媽時候,發現了秘密之所在,我只能說,科興科學園真是我的寶地啊。 涉及到的netty類:NioWorker、HeapChannelBufferFactory、BigEndianHeapChannelBuffer、SocketReceiveBufferPool 核心的秘密在SocketReceiveBufferPool中
1 final class SocketReceiveBufferPool { 2 3 private static final int POOL_SIZE = 8; 4 5 @SuppressWarnings("unchecked") 6 private final SoftReference<ByteBuffer>[] pool = new SoftReference[POOL_SIZE]; 7 8 SocketReceiveBufferPool() { 9 super(); 10 } 11 12 final ByteBuffer acquire(int size) { 13 final SoftReference<ByteBuffer>[] pool = this.pool; 14 for (int i = 0; i < POOL_SIZE; i ++) { 15 SoftReference<ByteBuffer> ref = pool[i]; 16 if (ref == null) { 17 continue; 18 } 19 20 ByteBuffer buf = ref.get(); 21 if (buf == null) { 22 pool[i] = null; 23 continue; 24 } 25 26 if (buf.capacity() < size) { 27 continue; 28 } 29 30 pool[i] = null; 31 32 buf.clear(); 33 return buf; 34 } 35 36 ByteBuffer buf = ByteBuffer.allocateDirect(normalizeCapacity(size)); 37 buf.clear(); 38 return buf; 39 } 40 41 final void release(ByteBuffer buffer) { 42 final SoftReference<ByteBuffer>[] pool = this.pool; 43 for (int i = 0; i < POOL_SIZE; i ++) { 44 SoftReference<ByteBuffer> ref = pool[i]; 45 if (ref == null || ref.get() == null) { 46 pool[i] = new SoftReference<ByteBuffer>(buffer); 47 return; 48 } 49 } 50 51 // pool is full - replace one 52 final int capacity = buffer.capacity(); 53 for (int i = 0; i< POOL_SIZE; i ++) { 54 SoftReference<ByteBuffer> ref = pool[i]; 55 ByteBuffer pooled = ref.get(); 56 if (pooled == null) { 57 pool[i] = null; 58 continue; 59 } 60 61 if (pooled.capacity() < capacity) { 62 pool[i] = new SoftReference<ByteBuffer>(buffer); 63 return; 64 } 65 } 66 } 67 68 private static final int normalizeCapacity(int capacity) { 69 // Normalize to multiple of 1024 70 int q = capacity >>> 10; 71 int r = capacity & 1023; 72 if (r != 0) { 73 q ++; 74 } 75 return q << 10; 76 } 77 }SocketReceiveBufferPool中維護了一個SoftReference<ByteBuffer>類型的數組,關於java的SoftReference,大家可以自行搜索。其實就是在此類中維護了一個directbuffer的記憶體池,此部分的記憶體是可以重覆利用的。那麼問題來了,如果我們把netty用於接收網路信息的directbuffer直接傳給dubbo的業務代碼,那麼這個記憶體池的作用是什麼呢,記憶體如何被release回記憶體池?帶著這個疑問,繼續分析調用了SocketReceiveBufferPool的NioWorker代碼。
1 private boolean read(SelectionKey k) { 2 final SocketChannel ch = (SocketChannel) k.channel(); 3 final NioSocketChannel channel = (NioSocketChannel) k.attachment(); 4 5 final ReceiveBufferSizePredictor predictor = 6 channel.getConfig().getReceiveBufferSizePredictor(); 7 final int predictedRecvBufSize = predictor.nextReceiveBufferSize(); 8 9 int ret = 0; 10 int readBytes = 0; 11 boolean failure = true; 12 13 ByteBuffer bb = recvBufferPool.acquire(predictedRecvBufSize); 14 15 try { 16 while ((ret = ch.read(bb)) > 0) { 17 readBytes += ret; 18 if (!bb.hasRemaining()) { 19 break; 20 } 21 } 22 failure = false; 23 } catch (ClosedChannelException e) { 24 // Can happen, and does not need a user attention. 25 } catch (Throwable t) { 26 fireExceptionCaught(channel, t); 27 } 28 29 if (readBytes > 0) { 30 bb.flip(); 31 32 final ChannelBufferFactory bufferFactory = 33 channel.getConfig().getBufferFactory(); 34 final ChannelBuffer buffer = bufferFactory.getBuffer(readBytes); 35 buffer.setBytes(0, bb); 36 buffer.writerIndex(readBytes); 37 //if(buffer instanceof BigEndianHeapChannelBuffer){ 38 // logger2.info("buffer instanceof BigEndianHeapChannelBuffer."); 39 //} 40 recvBufferPool.release(bb); 41 42 // Update the predi||\\||||| 43 predictor.previousReceiveBufferSize(readBytes); 44 45 // Fire the event. 46 fireMessageReceived(channel, buffer); 47 } else { 48 recvBufferPool.release(bb); 49 } 50 51 if (ret < 0 || failure) { 52 k.cancel(); // Some JDK implementations run into an infinite loop without this. 53 close(channel, succeededFuture(channel)); 54 return false; 55 } 56 57 return true; 58 }在代碼里發現了netty會再創造一個chanelbuffer對象,然後將directbuffer里的內容複製到chanelbuffer裡面,而這個chanelbuffer對象實際上是一個堆內記憶體,然後netty會真對這塊記憶體進行解碼及返回給上層調用服務等,也就是說沒有直接將directbuffer返回給dubbo服務,這樣也就解釋了,我們在提供dubbo服務的jvm里,禁止掉了system.gc的情況下,沒有發生過堆外記憶體泄漏的原因。後面我會找時間詳細的分析一下netty4和kafka使用directbuffer的情況。