一,原理及介紹〇 xtrabackup能做哪些 對InnoDB引擎的表做熱備 增量備份 流壓縮傳輸到另外的伺服器上 線上移動表 更簡單的創建從庫 備份時不增加伺服器負載 〇 原理 備份及恢復大致涉及三個步驟:備份 -> prepare -> 恢復 備份運行時,工具會記住當時的LSN號,並打開xtra ...
一,原理及介紹
〇 xtrabackup能做哪些
對InnoDB引擎的表做熱備
增量備份
流壓縮傳輸到另外的伺服器上
線上移動表
更簡單的創建從庫
備份時不增加伺服器負載
〇 原理
備份及恢復大致涉及三個步驟:備份 -> prepare -> 恢復
備份運行時,工具會記住當時的LSN號,並打開xtrabackup_logfile,然後開始對datafile進行copy,即ibdata1及ibd文件。
複製需要一定的時間,在複製期間,如果文件被修改,工具將監視redo log file並將每一次更變記錄下來,保存在xtrabackup_logfile中。
接下來處理非事務表如MyISAM的備份操作,innobackupex通過FLUSH TABLES WITH READ LOCK來阻塞DML。
併在此時獲取binlog的position[和GTID](此處我理解為和mysqldump --single-transaction處理方式類似)
在做完非事務表的copy之後,執行UNLOCK TABLES,完成備份,並停止記錄xtrabackup_logfile。
接下來就是需要做prepare的過程,該過程類似InnoDB的crash-recovery。
對redo log進行前滾(到數據文件),並將沒提交的事務進行回滾操作(rollback),這樣便可以保證數據的一致性,所以對於事務表,整個過程是不會影響寫操作的。
註:InnoDB、XtraDB、MyISAM是肯定支持的,其他的存儲引擎沒有測過。
〇 許可權需求
對datadir需要有rwx的許可權。
MySQL:
最小所需要的許可權有:
RELOAD
LOCK TABLES(如果加上--no-lock的話可以不要)
REPLICATION CLIENT(為了獲得binary log的position)
PROCESS(為了執行show engine innodb status,並且需要查看所有運行的線程)
其他可能需要用到的許可權:
CREATE TABLESPACE(如果需要通過5.6+ 的TTS恢復/遷移單個表的話)
SUPER(可能需要在複製環境里啟動或者停止slave線程)
CREATE\INSERT\SELECT(對PERCONA_SCHEMA.xtrabackup_history進行操作)
二、下載安裝
官方地址:https://www.percona.com/downloads/XtraBackup/LATEST/
安裝:
tar xf Percona-XtraBackup-2.4.12-r170eb8c-el7-x86_64-bundle.tar
rpm -vih percona-xtrabackup*.rpm 或者yum install -y percona-xtrabackup-*
三,全量備份與恢復
•備份數據
語法:innobackupex [--defaults-file=/etc/my.cnf] --user=DBUSER --password=DBUSERPASS [--socket=/var/lib/mysql/mysql.sock --port=3306] /path/to/BACKUP-DIR/

[root@Server data]# innobackupex --defaults-file=/etc/my.cnf --user=root --password=939879412 /data/ xtrabackup: recognized server arguments: --datadir=/var/lib/mysql --log_bin=server-bin.log --server-id=1 xtrabackup: recognized client arguments: --datadir=/var/lib/mysql --log_bin=server-bin.log --server-id=1 181029 14:41:49 innobackupex: Starting the backup operation IMPORTANT: Please check that the backup run completes successfully. At the end of a successful backup run innobackupex prints "completed OK!". 181029 14:41:50 version_check Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_group=xtrabackup' as 'root' (using passw ord: YES).181029 14:41:50 version_check Connected to MySQL server 181029 14:41:50 version_check Executing a version check against the server... 181029 14:41:50 version_check Done. 181029 14:41:50 Connecting to MySQL server host: localhost, user: root, password: set, port: not set, socket: not set Using server version 5.6.41 innobackupex version 2.4.12 based on MySQL server 5.7.19 Linux (x86_64) (revision id: 170eb8c) xtrabackup: uses posix_fadvise(). xtrabackup: cd to /var/lib/mysql xtrabackup: open files limit requested 0, set to 65535 xtrabackup: using the following InnoDB configuration: xtrabackup: innodb_data_home_dir = . xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend xtrabackup: innodb_log_group_home_dir = ./ xtrabackup: innodb_log_files_in_group = 2 xtrabackup: innodb_log_file_size = 50331648 InnoDB: Number of pools: 1 181029 14:41:50 >> log scanned up to (2048699761) xtrabackup: Generating a list of tablespaces InnoDB: Allocated tablespace ID 4 for mysql/slave_master_info, old maximum was 0 181029 14:41:51 [01] Copying ./ibdata1 to /data/2018-10-29_14-41-49/ibdata1 181029 14:41:51 >> log scanned up to (2048699761) 181029 14:41:52 >> log scanned up to (2048699952) 181029 14:41:53 >> log scanned up to (2048700648) 181029 14:41:54 >> log scanned up to (2048700949) 181029 14:41:55 >> log scanned up to (2048701233) 181029 14:41:56 >> log scanned up to (2048701233) 181029 14:41:57 >> log scanned up to (2048701533) 181029 14:41:58 >> log scanned up to (2048701928) ......... 181029 14:42:09 Finished backing up non-InnoDB tables and files 181029 14:42:09 Executing FLUSH NO_WRITE_TO_BINLOG ENGINE LOGS... xtrabackup: The latest check point (for incremental): '2048706153' xtrabackup: Stopping log copying thread. .181029 14:42:09 >> log scanned up to (2048706153) 181029 14:42:09 Executing UNLOCK TABLES 181029 14:42:09 All tables unlocked 181029 14:42:09 Backup created in directory '/data/2018-10-29_14-41-49/' 181029 14:42:09 [00] Writing /data/2018-10-29_14-41-49/backup-my.cnf 181029 14:42:09 [00] ...done 181029 14:42:09 [00] Writing /data/2018-10-29_14-41-49/xtrabackup_info 181029 14:42:09 [00] ...done xtrabackup: Transaction log of lsn (2048670711) to (2048706153) was copied. 181029 14:42:10 completed OK!View Code
當出現innobackupex: completed OK!
出現上面的信息,表示備份已經ok。
查看備份的結果
[root@Server data]# ll total 4 drwxr-x--- 7 root root 4096 Oct 29 14:42 2018-10-29_14-41-49 [root@Server data]# cd 2018-10-29_14-41-49/ [root@Server 2018-10-29_14-41-49]# ls backup-my.cnf ibdata1 mysql performance_schema test test1 xtrabackup_checkpoints xtrabackup_info xtrabackup_logfile zabbix
將資料庫備份到遠程目錄
前提條件:
a.遠程機器的目錄要存在,同時保證是空目錄。遠程機器目錄非空,備份時報以下錯誤"xtrabackup: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)"
b.ssh要能連接:使用ssh秘鑰免密碼輸入
innobackupex --user=USER --password=PASSWD --socket=/var/lib/mysql/mysql.sock --port=3306 --stream=tar /data/temp/ | gzip | ssh root@遠程主機host "cat -> /data/temp/mysql_full_backup_2018-10-29.tar.gz"
在備份的文件夾中,有幾個文件值得註意:
xtrabackup_binlog_pos_innodb記錄了binlog的position,若開啟了GTID,也會將GTID取出。
在用於備份+binlog恢復或建立slave的場景里十分有用。
xtrabackup_checkpoints記錄了此次備份的類型和lsn號的起始值,是否壓縮等
xtrabackup_info則記錄了備份工具的信息,時間,備份對象(是針對全實例還是某庫表),是否是增量,binlog位置等
backup-my.cnf文件,則記錄了備份時可能涉及到的選項參數,比如系統表空間信息,獨立undo表空間信息,redo-log信息等
• 恢複數據
註意:恢復之前
1)要先關閉資料庫
2)要刪除數據文件和日誌文件(也可以mv移到別的地方,只要確保清空mysql數據存放目錄就行)
apply-log 類似innodb的crash recovery

[root@Server data]# innobackupex --apply-log /data/2018-10-29_14-41-49/ #prepare過程 xtrabackup: recognized server arguments: --innodb_checksum_algorithm=innodb --innodb_log_checksum_algorithm=innodb --innodb_data_file_path= ibdata1:12M:autoextend --innodb_log_files_in_group=2 --innodb_log_file_size=50331648 --innodb_fast_checksum=0 --innodb_page_size=16384 --innodb_log_block_size=512 --innodb_undo_directory=. --innodb_undo_tablespaces=0 --server-id=1 --redo-log-version=0 xtrabackup: recognized client arguments: --innodb_checksum_algorithm=innodb --innodb_log_checksum_algorithm=innodb --innodb_data_file_path= ibdata1:12M:autoextend --innodb_log_files_in_group=2 --innodb_log_file_size=50331648 --innodb_fast_checksum=0 --innodb_page_size=16384 --innodb_log_block_size=512 --innodb_undo_directory=. --innodb_undo_tablespaces=0 --server-id=1 --redo-log-version=0 181029 14:50:05 innobackupex: Starting the apply-log operation IMPORTANT: Please check that the apply-log run completes successfully. At the end of a successful apply-log run innobackupex prints "completed OK!". innobackupex version 2.4.12 based on MySQL server 5.7.19 Linux (x86_64) (revision id: 170eb8c) xtrabackup: cd to /data/2018-10-29_14-41-49/ xtrabackup: This target seems to be not prepared yet. InnoDB: Number of pools: 1 xtrabackup: xtrabackup_logfile detected: size=8388608, start_lsn=(2048670711) xtrabackup: using the following InnoDB configuration for recovery: xtrabackup: innodb_data_home_dir = . xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend xtrabackup: innodb_log_group_home_dir = . xtrabackup: innodb_log_files_in_group = 1 xtrabackup: innodb_log_file_size = 8388608 xtrabackup: using the following InnoDB configuration for recovery: xtrabackup: innodb_data_home_dir = . xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend xtrabackup: innodb_log_group_home_dir = . xtrabackup: innodb_log_files_in_group = 1 xtrabackup: innodb_log_file_size = 8388608 xtrabackup: Starting InnoDB instance for recovery. xtrabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter) InnoDB: PUNCH HOLE support available InnoDB: Mutexes and rw_locks use GCC atomic builtins InnoDB: Uses event mutexes InnoDB: GCC builtin __sync_synchronize() is used for memory barrier InnoDB: Compressed tables use zlib 1.2.3 InnoDB: Number of pools: 1 InnoDB: Using CPU crc32 instructions InnoDB: Initializing buffer pool, total size = 100M, instances = 1, chunk size = 100M InnoDB: Completed initialization of buffer pool InnoDB: page_cleaner coordinator priority: -20 InnoDB: Highest supported file format is Barracuda. InnoDB: Log scan progressed past the checkpoint lsn 2048670711 InnoDB: Doing recovery: scanned up to log sequence number 2048706153 (0%) InnoDB: Database was not shutdown normally! InnoDB: Starting crash recovery. InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percent: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: xtrabackup: Last MySQL binlog file position 11989, file name ./server-bin.000044 InnoDB: Creating shared tablespace for temporary tables InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... InnoDB: File './ibtmp1' size is now 12 MB. InnoDB: 96 redo rollback segment(s) found. 1 redo rollback segment(s) are active. InnoDB: 32 non-redo rollback segment(s) are active. InnoDB: Waiting for purge to start InnoDB: 5.7.19 started; log sequence number 2048706153 InnoDB: xtrabackup: Last MySQL binlog file position 11989, file name ./server-bin.000044 xtrabackup: starting shutdown with innodb_fast_shutdown = 1 InnoDB: FTS optimize thread exiting. InnoDB: Starting shutdown... InnoDB: Shutdown completed; log sequence number 2048707915 InnoDB: Number of pools: 1 xtrabackup: using the following InnoDB configuration for recovery: xtrabackup: innodb_data_home_dir = . xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend xtrabackup: innodb_log_group_home_dir = . xtrabackup: innodb_log_files_in_group = 2 xtrabackup: innodb_log_file_size = 50331648 InnoDB: PUNCH HOLE support available InnoDB: Mutexes and rw_locks use GCC atomic builtins InnoDB: Uses event mutexes InnoDB: GCC builtin __sync_synchronize() is used for memory barrier InnoDB: Compressed tables use zlib 1.2.3 InnoDB: Number of pools: 1 InnoDB: Using CPU crc32 instructions InnoDB: Initializing buffer pool, total size = 100M, instances = 1, chunk size = 100M InnoDB: Completed initialization of buffer pool InnoDB: page_cleaner coordinator priority: -20 InnoDB: Setting log file ./ib_logfile101 size to 48 MB InnoDB: Setting log file ./ib_logfile1 size to 48 MB InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0 InnoDB: New log files created, LSN=2048707915 InnoDB: Highest supported file format is Barracuda. InnoDB: Log scan progressed past the checkpoint lsn 2048708108 InnoDB: Doing recovery: scanned up to log sequence number 2048708117 (0%) InnoDB: Database was not shutdown normally! InnoDB: Starting crash recovery. InnoDB: xtrabackup: Last MySQL binlog file position 11989, file name ./server-bin.000044 InnoDB: Removed temporary tablespace data file: "ibtmp1" InnoDB: Creating shared tablespace for temporary tables InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... InnoDB: File './ibtmp1' size is now 12 MB. InnoDB: 96 redo rollback segment(s) found. 1 redo rollback segment(s) are active. InnoDB: 32 non-redo rollback segment(s) are active. InnoDB: Waiting for purge to start InnoDB: 5.7.19 started; log sequence number 2048708117 xtrabackup: starting shutdown with innodb_fast_shutdown = 1 InnoDB: FTS optimize thread exiting. InnoDB: Starting shutdown... InnoDB: Shutdown completed; log sequence number 2048713467 181029 14:50:09 completed OK!View Code
當出現innobackupex: completed OK!
出現上面的信息,表示恢復已經ok。
# cp -r 2018-10-29_14-41-49/* /var/lib/mysql/ 將數據拷貝到資料庫的數據目錄下
也可以用此命令拷貝 innobackupex --copy-back /var/lib/mysql/
# chown mysql:mysql /var/lib/mysql/ -R
重啟資料庫
四,增量備份與恢復
①增備方法與全備不一樣:
innobackupex --user= --password= --incremental $new_dir --incremental-basedir=$basedir
其中--incremental是本次增量備份存放目錄
$new_dir是表示將增量備份出來的東西放在哪個目錄
--incremental-basedir則表示,針對哪一次備份做增量備份
備份的差異在目錄的xtrabackup_checkpoints中查看:
比如:
$basedir中內容:
backup_type = full-prepared
from_lsn = 0
to_lsn = 2048706153
last_lsn = 2048706153
compact = 0
recover_binlog_info = 0
$new_bkdir中內容:
backup_type = incremental
from_lsn = 2048706153
to_lsn = 2048876543
last_lsn = 2048876543
compact = 0
recover_binlog_info = 0
可以註意一下增備的from_lsn號
大於這個LSN號的頁都是被變更過的,這些偏移量,也就是需要被增量備份出去的
②prepare:
prepare過程:
從第一個備份開始(也就是全量)做prepare,再將往後的增量備份依次添加到全量備份中。
註意,此處多了一個參數即--redo-only,該參數是指將已提交的事務應用,未提交的事務回滾。
此外,--incremental-dir也是在之前沒有用到過的,這個參數代表需要被合併進去的增量備份目錄。
註意,此處多次的增量備份是指:針對上次的增量備份做的增量。
也就是可以理解為:
全備:500GB
第一次增量備份:2GB
第二次增量備份:1GB(針對第一次增量備份的增量數據)
……
第n次
按照備份順序做prepare,也就是prepare的順序為:
第一次全備 -> 增量備份1 -> 增量備份2 -> ... -> 增量備份n
第一次全備的prepare:innobackup --apply-log --redo-only $basedir
第二次prepare:innobackup --apply-log --redo-only $basedir --incremental-dir=$new_dir_1(此處的$new_dir_1也就是第一次增量備份)
......
第n次prepare:innobackup --apply-log $basedir --incremental-dir=$new_dir_n(此處的$new_dir_n也就是最近也就是最後一次的增量備份
最後一次增量備份的prepare,不需要指定--redo-only
最後將增量備份和全備進行合併,將未提交的事務回滾,這個操作和全量prepare無異:
innobackup --apply-log $basedir
③恢復到datadir:
和全量無異,直接copyback就行了
innobackupex --copy-back $basedir
增量備份的prepare有點特殊,還是小結一下:
① prepare完備(加上--redo-only)
② prepare每一次增量備份到完備中,需要加上--redo-only,最後一次增量備份的prepare不需要加--redo-only
③ 對生成的最終完備做--apply-log
執行第一次增量備份之後,可以再做一次增量備份
此時有兩種增量備份方法:
第一種,總是針對basedir做增量,這個方式恢復起來就特別簡單了,只需要將最後一次的增量備份合併到全量備份里,就可以恢復了。
第二種,總是針對上一次的增量,做增量備份。這個方式的恢復,就要逐一合併了,也就是我上述所說看起來有點複雜的增備思路。
圖示:
第一種:
總是將1月1日的全備作為basedir,所以FROM_LSN號總是5000。
第二種:
總是把上一次(最近一次)的備份作為basedir。