前言 今天這篇博客是接我的上一篇博客 https://www.cnblogs.com/y-yp/p/12191258.html,繼續介紹一下MMF在Linux上的用法 ps:本來本地調試完case,想放到伺服器上跑跑看,結果竟然報"PlatformNotSupportedException",然後仔 ...
前言
今天這篇博客是接我的上一篇博客 https://www.cnblogs.com/y-yp/p/12191258.html,繼續介紹一下MMF在Linux上的用法
ps:本來本地調試完case,想放到伺服器上跑跑看,結果竟然報"PlatformNotSupportedException",然後仔細一查,竟然發現MMF在Windows和Linux上的用法不一樣。。。"mapName"參數僅作為Window平臺的一個特性,在Linux平臺上只能傳"null",於是就有了今天這篇博客
實現
既然“mapName”不能使用,經過測試選定使用了FileStream的這個重載
具體細節就不在介紹了,有疑問的話可以參考我的上一篇博客 https://www.cnblogs.com/y-yp/p/12191258.html,這裡直接給實現
先定義“行數據信息記錄“,這個用來讀取數據的時候用,一行數據只生成一條記錄,所以在大數據量的情況下也不會占用很多記憶體
public class RowInfo { /// <summary> /// 行數據體積(單位位元組) /// </summary> public long Capacity { get; set; } /// <summary> /// 行單元格個數 /// </summary> public int CellQuantity { get; set; } }
然後開始將數據寫入MMF文件,並獲取到”行數據信息記錄“
//準備數據 var data = new List<string[]>(); for (var i = 0; i < 100; i++) { var rowData = new string[100]; for (var j = 0; j < 100; j++) { rowData[j] = $"{i}-{j}"; } data.Add(rowData); } //統計mmf文件體積,包含單元格數據的體積Encoding.UTF8.GetBytes(x).Length和預設單元格數據長度int類型占4位元組 var mmfCapacity = data.Sum(x => x.Sum(x => Encoding.UTF8.GetBytes(x).Length + 4)); var path = Environment.CurrentDirectory + "\\" + "test.txt"; using var writerFs = new FileStream(path, FileMode.Create, FileAccess.ReadWrite); using var writerMMF = MemoryMappedFile.CreateFromFile(writerFs, null, mmfCapacity, MemoryMappedFileAccess.ReadWrite, HandleInheritability.Inheritable, true); //記錄行數據信息 var rowInfos = new List<RowInfo>(); var totalWriterOffset = 0; foreach (var rowData in data) { var rowBuffers = rowData.Select(x => Encoding.UTF8.GetBytes(x)).ToList(); //計算行數據總體積 var capacity = rowBuffers.Sum(x => x.Length + 4); //根據當前偏移和需要讀取的長度創建accessor using var accessor = writerMMF.CreateViewAccessor(totalWriterOffset, capacity); //統計同行內單元格偏移 var offset = 0L; foreach (var cellBuffer in rowBuffers) { if (cellBuffer.Length > 0) { var dataSize = cellBuffer.Length; accessor.Write(offset, dataSize); accessor.WriteArray(offset + 4, cellBuffer, 0, dataSize); offset += 4 + dataSize; } else { accessor.Write(offset, 0); offset += 4; } } //記錄行數據信息 var rowInfo = new RowInfo() { Capacity = capacity, CellQuantity = rowBuffers.Count() }; rowInfos.Add(rowInfo); //總位移向前走一行數據的長度 totalWriterOffset += capacity; } return rowInfos;
通過”行數據信息記錄“還原數據,這裡可以將讀取出來的數據寫入自己的excel或者是csv文件,不再贅述
var result = new List<string[]>(); var path = Environment.CurrentDirectory + "\\" + "test.txt"; //從行數據信息記錄統計mmf文件總體積 var mmfCapacity = rowInfos.Sum(x => x.Capacity); var totalReaderOffset = 0; using var readerFs = new FileStream(path, FileMode.Open, FileAccess.ReadWrite); using var readerMMF = MemoryMappedFile.CreateFromFile(readerFs, null, mmfCapacity, MemoryMappedFileAccess.ReadWrite, HandleInheritability.Inheritable, true); foreach(var rowInfo in rowInfos) { var rowData = new string[rowInfo.CellQuantity]; using var accessor = readerMMF.CreateViewAccessor(totalReaderOffset, rowInfo.Capacity); var position = 0; for (int cellIndex = 0; cellIndex < rowInfo.CellQuantity; cellIndex++) { var cellSize = accessor.ReadInt32(position); var buffer = new byte[cellSize]; accessor.ReadArray(position + 4, buffer, 0, cellSize); rowData[cellIndex] = Encoding.UTF8.GetString(buffer); position += 4 + cellSize; } result.Add(rowData); } if (File.Exists(path)) { File.Delete(path); }
考慮使用記憶體映射文件的話,可以先本地測試一下性能,如果是SSD的話性能還是很不錯的,綜合跑下來跟直接寫入記憶體速度相差不會超過一兩倍(記憶體使用率較高的話會嚴重降低性能,甚至會OOM),而且這其中還有很多的優化空間
今天的文章只是提了個思路,細節還有很多要考慮,有疑問的話歡迎提問交流~~