MySQL 8.0 Reference Manual(讀書筆記79節-- InnoDB Table and Page Compression (2))

Play Games

華為雲GeminiDB是一款相容Redis協議的彈性KV資料庫,支持遠超記憶體的容量和極致的性能,技術自主創新,不受Redis協議變更影響。 ...

1 Monitoring InnoDB Table Compression at Runtime

Overall application performance, CPU and I/O utilization and the size of disk files are good indicators of how effective compression is for your application. This section builds on the performance tuning advice and shows how to find problems that might not turn up during initial testing.

To dig deeper into performance considerations for compressed tables, you can monitor compression performance at runtime using the Information Schema tables.These tables reflect the internal use of memory and the rates of compression used overall.


The INNODB_CMP table reports information about compression activity for each compressed page size (KEY_BLOCK_SIZE) in use. The information in these tables is system-wide: it summarizes the compression statistics across all compressed tables in your database. You can use this data to help decide whether or not to compress a table by examining these tables when no other compressed tables are being accessed. It involves relatively low overhead on the server, so you might query it periodically on a production server to check the overall efficiency of the compression feature. 


The INNODB_CMP_PER_INDEX table reports information about compression activity for individual tables and indexes. This information is more targeted and more useful for evaluating compression efficiency and diagnosing performance issues one table or index at a time. (Because that each InnoDB table is represented as a clustered index, MySQL does not make a big distinction between tables and indexes in this context.) The INNODB_CMP_PER_INDEX table does involve substantial overhead, so it is more suitable for development servers, where you can compare the effects of different workloads, data, and compression settings in isolation. To guard against imposing this monitoring overhead by accident, you must enable the innodb_cmp_per_index_enabled configuration option before you can query the INNODB_CMP_PER_INDEX table.


The key statistics to consider are the number of, and amount of time spent performing, compression and uncompression operations. Since MySQL splits B-tree nodes when they are too full to contain the compressed data following a modification, compare the number of “successful” compression operations with the number of such operations overall. Based on the information in the INNODB_CMP and INNODB_CMP_PER_INDEX tables and overall application performance and hardware resource utilization, you might make changes in your hardware configuration, adjust the size of the buffer pool, choose a different page size, or select a different set of tables to compress.

If the amount of CPU time required for compressing and uncompressing is high, changing to faster or multi-core CPUs can help improve performance with the same data, application workload and set of compressed tables. Increasing the size of the buffer pool might also help performance, so that more uncompressed pages can stay in memory, reducing the need to uncompress pages that exist in memory only in compressed form.

A large number of compression operations overall (compared to the number of INSERT, UPDATE and DELETE operations in your application and the size of the database) could indicate that some of your compressed tables are being updated too heavily for effective compression. If so, choose a larger page size, or be more selective about which tables you compress.

If the number of “successful” compression operations (COMPRESS_OPS_OK) is a high percentage of the total number of compression operations (COMPRESS_OPS), then the system is likely performing well. If the ratio is low, then MySQL is reorganizing, recompressing, and splitting B-tree nodes more often than is desirable. In this case, avoid compressing some tables, or increase KEY_BLOCK_SIZE for some of the compressed tables. You might turn off compression for tables that cause the number of “compression failures” in your application to be more than 1% or 2% of the total. (Such a failure ratio might be acceptable during a temporary operation such as a data load).

2. How Compression Works for InnoDB Tables --工作原理

 This section describes some internal implementation details about compression for InnoDB tables. The information presented here may be helpful in tuning for performance, but is not necessary to know for basic use of compression.

2.1 Compression Algorithms

Some operating systems implement compression at the file system level. Files are typically divided into fixed-size blocks that are compressed into variable-size blocks, which easily leads into fragmentation【fræɡmenˈteɪʃn 碎片;碎裂(化),片段,摘錄】. Every time something inside a block is modified, the whole block is recompressed before it is written to disk. These properties make this compression technique unsuitable for use in an update-intensive database system.

MySQL implements compression with the help of the well-known zlib library, which implements the LZ77 compression algorithm. This compression algorithm is mature【məˈtʃʊr 成熟;充分發展;釀成;有判斷力】, robust【roʊˈbʌst 強健的;堅固的;強勁的;耐用的;結實的;強壯的;堅定的;富有活力的】, and efficient in both CPU utilization and in reduction of data size. The algorithm is “lossless”, so that the original uncompressed data can always be reconstructed from the compressed form. LZ77 compression works by finding sequences of data that are repeated within the data to be compressed. The patterns of values in your data determine how well it compresses, but typical user data often compresses by 50% or more.

補充:InnoDB supports the zlib library up to version 1.2.11, which is the version bundled with MySQL 8.0.

Unlike compression performed by an application, or compression features of some other database management systems, InnoDB compression applies both to user data and to indexes. In many cases, indexes can constitute 40-50% or more of the total database size, so this difference is significant. When compression is working well for a data set, the size of the InnoDB data files (the file-per-table tablespace or general tablespace .ibd files) is 25% to 50% of the uncompressed size or possibly smaller【如果條件好的話,壓縮後,是原來的 1/4 ~1/2 】. Depending on the workload, this smaller database can in turn lead to a reduction in I/O, and an increase in throughput, at a modest cost in terms of increased CPU utilization. You can adjust the balance between compression level and CPU overhead by modifying the innodb_compression_level configuration option.

2.2 InnoDB Data Storage and Compression

All user data in InnoDB tables is stored in pages comprising a B-tree index (the clustered index). In some other database systems, this type of index is called an “index-organized table”. Each row in the index node contains the values of the (user-specified or system-generated) primary key and all the other columns of the table.

Secondary indexes in InnoDB tables are also B-trees, containing pairs of values: the index key and a pointer to a row in the clustered index. The pointer is in fact the value of the primary key of the table, which is used to access the clustered index if columns other than the index key and primary key are required. Secondary index records must always fit on a single B-tree page.

The compression of B-tree nodes (of both clustered and secondary indexes) is handled differently from compression of overflow pages used to store long VARCHAR, BLOB, or TEXT columns.

2.2.1 Compression of B-Tree Pages

Because they are frequently updated, B-tree pages require special treatment. It is important to minimize the number of times B-tree nodes are split, as well as to minimize the need to uncompress and recompress their content.

One technique MySQL uses is to maintain some system information in the B-tree node in uncompressed form, thus facilitating certain in-place updates. For example, this allows rows to be delete-marked and deleted without any compression operation.

In addition, MySQL attempts to avoid unnecessary uncompression and recompression of index pages when they are changed. Within each B-tree page, the system keeps an uncompressed “modification log” to record changes made to the page. Updates and inserts of small records may be written to this modification log without requiring the entire page to be completely reconstructed.

When the space for the modification log runs out, InnoDB uncompresses the page, applies the changes and recompresses the page. If recompression fails (a situation known as a compression failure), the B-tree nodes are split and the process is repeated until the update or insert succeeds.

To avoid frequent compression failures in write-intensive workloads, such as for OLTP applications, MySQL sometimes reserves【rɪˈzɜːrvz 保留;擁有,保持】 some empty space (padding) in the page, so that the modification log fills up sooner and the page is recompressed while there is still enough room to avoid splitting it. The amount of padding space left in each page varies as the system keeps track of the frequency of page splits. On a busy server doing frequent writes to compressed tables, you can adjust the innodb_compression_failure_threshold_pct, and innodb_compression_pad_pct_max configuration options to fine-tune this mechanism.

Generally, MySQL requires that each B-tree page in an InnoDB table can accommodate【əˈkɑːmədeɪt 容納;順應,適應(新情況);考慮到;顧及;幫忙;為…提供空間;為(某人)提供住宿(或膳宿、座位等)】 at least two records. For compressed tables, this requirement has been relaxed. Leaf pages of B-tree nodes (whether of the primary key or secondary indexes) only need to accommodate one record, but that record must fit, in uncompressed form, in the per-page modification log. If innodb_strict_mode is ON, MySQL checks the maximum row size during CREATE TABLE or CREATE INDEX. If the row does not fit, the following error message is issued: ERROR HY000: Too big row.

If you create a table when innodb_strict_mode is OFF, and a subsequent INSERT or UPDATE statement attempts to create an index entry that does not fit in the size of the compressed page, the operation fails with ERROR 42000: Row size too large. (This error message does not name the index for which the record is too large, or mention the length of the index record or the maximum record size on that particular index page.) To solve this problem, rebuild the table with ALTER TABLE and select a larger compressed page size (KEY_BLOCK_SIZE), shorten any column prefix indexes, or disable compression entirely with ROW_FORMAT=DYNAMIC or ROW_FORMAT=COMPACT.

innodb_strict_mode is not applicable to general tablespaces, which also support compressed tables. Tablespace management rules for general tablespaces are strictly enforced independently of innodb_strict_mode. 

2.2.2 Compressing BLOB, VARCHAR, and TEXT Columns

In an InnoDB table, BLOB, VARCHAR, and TEXT columns that are not part of the primary key may be stored on separately allocated overflow pages. We refer to these columns as off-page columns. Their values are stored on singly-linked lists of overflow pages.

For tables created in ROW_FORMAT=DYNAMIC or ROW_FORMAT=COMPRESSED, the values of BLOB, TEXT, or VARCHAR columns may be stored fully off-page, depending on their length and the length of the entire row. For columns that are stored off-page, the clustered index record only contains 20-byte pointers to the overflow pages, one per column. Whether any columns are stored off-page depends on the page size and the total size of the row. When the row is too long to fit entirely within the page of the clustered index, MySQL chooses the longest columns for off-page storage until the row fits on the clustered index page. As noted above, if a row does not fit by itself on a compressed page, an error occurs.

補充:For tables created in ROW_FORMAT=DYNAMIC or ROW_FORMAT=COMPRESSED, TEXT and BLOB columns that are less than or equal to 40 bytes are always stored in-line.

Tables that use ROW_FORMAT=REDUNDANT and ROW_FORMAT=COMPACT store the first 768 bytes of BLOB, VARCHAR, and TEXT columns in the clustered index record along with the primary key. The 768-byte prefix is followed by a 20-byte pointer to the overflow pages that contain the rest of the column value.

When a table is in COMPRESSED format, all data written to overflow pages is compressed “as is”; that is, MySQL applies the zlib compression algorithm to the entire data item. Other than the data, compressed overflow pages contain an uncompressed header and trailer comprising a page checksum and a link to the next overflow page, among other things. Therefore, very significant storage savings can be obtained for longer BLOB, TEXT, or VARCHAR columns if the data is highly compressible, as is often the case with text data. Image data, such as JPEG, is typically already compressed and so does not benefit much from being stored in a compressed table; the double compression can waste CPU cycles for little or no space savings.

The overflow pages are of the same size as other pages. A row containing ten columns stored off-page occupies ten overflow pages, even if the total length of the columns is only 8K bytes. In an uncompressed table, ten uncompressed overflow pages occupy 160K bytes. In a compressed table with an 8K page size, they occupy only 80K bytes. Thus, it is often more efficient to use compressed table format for tables with long column values

For file-per-table tablespaces, using a 16K compressed page size can reduce storage and I/O costs for BLOB, VARCHAR, or TEXT columns, because such data often compress well, and might therefore require fewer overflow pages, even though the B-tree nodes themselves take as many pages as in the uncompressed form. General tablespaces do not support a 16K compressed page size (KEY_BLOCK_SIZE).

2.3 Compression and the InnoDB Buffer Pool

In a compressed InnoDB table, every compressed page (whether 1K, 2K, 4K or 8K) corresponds to an uncompressed page of 16K bytes (or a smaller size if innodb_page_size is set). To access the data in a page, MySQL reads the compressed page from disk if it is not already in the buffer pool, then uncompresses the page to its original form. This section describes how InnoDB manages the buffer pool with respect to pages of compressed tables.

To minimize I/O and to reduce the need to uncompress a page, at times the buffer pool contains both the compressed and uncompressed form of a database page. To make room for other required database pages, MySQL can evict from the buffer pool an uncompressed page, while leaving the compressed page in memory. Or, if a page has not been accessed in a while, the compressed form of the page might be written to disk, to free space for other data. Thus, at any given time, the buffer pool might contain both the compressed and uncompressed forms of the page, or only the compressed form of the page, or neither.

MySQL keeps track of which pages to keep in memory and which to evict using a least-recently-used (LRU) list, so that hot (frequently accessed) data tends to stay in memory. When compressed tables are accessed, MySQL uses an adaptive【əˈdæptɪv 適應的;有適應能力的】 LRU algorithm to achieve【/əˈtʃiːv 實現;(憑長期努力)達到(某目標、地位、標準);完成;成功】 an appropriate balance of compressed and uncompressed pages in memory. This adaptive algorithm is sensitive to whether the system is running in an I/O-bound or CPU-bound manner. The goal is to avoid spending too much processing time uncompressing pages when the CPU is busy, and to avoid doing excess I/O when the CPU has spare cycles that can be used for uncompressing compressed pages (that may already be in memory). When the system is I/O-bound, the algorithm prefers to evict the uncompressed copy of a page rather than both copies, to make more room for other disk pages to become memory resident. When the system is CPU-bound【baʊnd 跳躍;蹦跳】, MySQL prefers to evict 【ɪˈvɪkt 驅逐;逐出;】both the compressed and uncompressed page, so that more memory can be used for “hot” pages and reducing the need to uncompress data in memory only in compressed form.

2.4 Compression and the InnoDB Redo Log Files

Before a compressed page is written to a data file, MySQL writes a copy of the page to the redo log (if it has been recompressed since the last time it was written to the database). This is done to ensure that redo logs are usable for crash recovery, even in the unlikely case that the zlib library is upgraded and that change introduces a compatibility problem with the compressed data. Therefore, some increase in the size of log files, or a need for more frequent checkpoints, can be expected when using compression. The amount of increase in the log file size or checkpoint frequency depends on the number of times compressed pages are modified in a way that requires reorganization and recompression.

To create a compressed table in a file-per-table tablespace, innodb_file_per_table must be enabled. There is no dependence on the innodb_file_per_table setting when creating a compressed table in a general tablespace.




Play Games
  • 概述:通過對數組進行排序,代碼更好地利用了緩存,從而提高了程式的性能。這種現象通常被稱為"緩存友好"(cache-friendly)或"空間局部性"(spatial locality) 今天做一個數組數據計算時,發現一個效率問題,給大家分享一下 一個數組排序和不排序時同樣的邏輯處理速度是不一樣的。排 ...
  • // Stream MS HelpManual: // FileStream 官方手冊: ...
  • Grain 是 Orleans 框架中的基本單元,代表了應用程式中的一個實體或者一個計算單元。 每個Silo都是一個獨立的進程,Silo負責載入、管理和執行Grain實例,並處理來自客戶端的請求以及與其他Silo之間的通信。 通信原理 在相同的Silo中,Grain與Grain之間的通信通過直接的方 ...
  • SystemEvents 是一個開發 win32 視窗項目很常用的類,其中封裝了一些常用的系統廣播消息。在 WinUI3 項目中,SystemEvents 事件經常無法觸發,簡單排查了一下原因。 SystemEvent 內封裝了一個線程和一個視窗,通過視窗消息在內部線程上調用事件,內部使用了 Sys ...
  • 大家好,我是痞子衡,是正經搞技術的痞子。今天痞子衡給大家介紹的是i.MXRT1xxx系列GPIO提早供電會影響上電時序導致內部DCDC啟動失敗。 最近有一個 RW612 產品線的同事在設計一個雙 MCU 系統 Demo 時發現,當 RW612 板卡和 RT1060 板卡通過 UART 對接時,如果 ...
  • 目錄微型電腦的硬體共性結構及基本性能指標關於存儲器的介紹微型電腦的基本性能指標1. 字長2. 主頻3. 存儲容量4. 外設擴展能力5. 軟體配置情況Arm Cortex 系列微處理器系列概述Arm Cortex-A 系列處理器Arm Cortex-R 系列處理器Arm Cortex-M 系列處理 ...
  • 目錄遠程策略配置啟用遠程桌面使用設置啟用遠程桌面使用控制面板啟用遠程桌面 工作中有時需要使用遠程桌面,但工控機上面的策略一般都比較保守,遠程桌面經常會失敗。這裡記錄一下使用的遠程策略配置,方便以後工作中使用。 遠程策略配置 運行命令 gpedit.msc 打開本地策略編輯: 打開 電腦配置->管理 ...
  • 參考 Fedora Quick Docs Fedora Server Documentation Deploy an ARM64 Fedora VM on your PC: 3 steps Architectures/AArch64/Install with QEMU Virtualization ...
    Play Games
  • 不廢話,直接代碼 private Stack<Action> actionStack = new Stack<Action>(); private void SetCellValues() { var worksheet = Globals.ThisAddIn.Application.ActiveS ...
  • OpenAPI 規範是用於描述 HTTP API 的標準。該標準允許開發人員定義 API 的形狀,這些 API 可以插入到客戶端生成器、伺服器生成器、測試工具、文檔等中。儘管該標準具有普遍性和普遍性,但 ASP.NET Core 在框架內預設不提供對 OpenAPI 的支持。 當前 ASP.NET ...
  • @DateTimeFormat 和 @JsonFormat 是 Spring 和 Jackson 中用於處理日期時間格式的註解,它們有不同的作用: @DateTimeFormat @DateTimeFormat 是 Spring 框架提供的註解,用於指定字元串如何轉換為日期時間類型,以及如何格式化日 ...
  • 一、背景說明 1.1 效果演示 用python開發的爬蟲採集軟體,可自動抓取抖音評論數據,並且含二級評論! 為什麼有了源碼還開發界面軟體呢?方便不懂編程代碼的小白用戶使用,無需安裝python、無需懂代碼,雙擊打開即用! 軟體界面截圖: 爬取結果截圖: 以上。 1.2 演示視頻 軟體運行演示視頻:見 ...
  • SpringBoot筆記 SpringBoot文檔 官網: 學習文檔: 線上API: http ...
  • 作為後端工程師,多數情況都是給別人提供介面,寫的好不好使你得重視起來。 最近我手頭一些活,需要和外部公司對接,我們需要提供一個介面文檔,這樣可以節省雙方時間、也可以防止後續扯皮。這是就要考驗我的介面是否規範化。 1. 介面名稱清晰、明確 顧名思義,介面是做什麼的,是否準確、清晰?讓使用這一眼就能知道 ...
  • 本文介紹基於Python語言,遍歷文件夾並從中找到文件名稱符合我們需求的多個.txt格式文本文件,並從上述每一個文本文件中,找到我們需要的指定數據,最後得到所有文本文件中我們需要的數據的合集的方法~ ...
  • Java JUC&多線程 基礎完整版 目錄Java JUC&多線程 基礎完整版1、 多線程的第一種啟動方式之繼承Thread類2、多線程的第二種啟動方式之實現Runnable介面3、多線程的第三種實現方式之實現Callable介面4、多線的常用成員方法5、線程的優先順序6、守護線程7、線程的讓出8、線 ...
  • 實時識別關鍵詞是一種能夠將搜索結果提升至新的高度的API介面。它可以幫助我們更有效地分析文本,並提取出關鍵詞,以便進行進一步的處理和分析。 該介面是挖數據平臺提供的,有三種模式:精確模式、全模式和搜索引擎模式。不同的模式在分詞的方式上有所不同,適用於不同的場景。 首先是精確模式。這種模式會儘量將句子 ...
  • 1 為啥要折騰搭建一個專屬圖床? 技術大佬寫博客都用 md 格式,要在多平臺發佈,圖片就得有外鏈 後續如博客遷移,國內博客網站如掘金,簡書,語雀等都做了防盜鏈,圖片無法遷移 2 為啥選擇CloudFlare R2 跳轉: 有白嫖額度 免費 CD ...