一、文檔說明 二、操作記錄 --第一次導入操作failed --失敗原因,導入表創建語法DDL失敗,原因 1.導入環境為XE學習環境,資料庫容量最大11g,如果空間足夠,不會報錯 2.導入建表語法,dump文件小於200M,導入建表語句還未插入數據,已占用11G存儲,由於表的STORAGE(INIT ...
一、文檔說明
1.0 需求:需要將生產環境PICC用戶導出,在測試環境中docker 測試資料庫導入,只需要表結構;
2.0 思路:根據開發人員提供的需求,資料庫源端aix 11.2.0.4, 目標端11.2.0.3
工具:使用expdp進行導出,impdp進行導入,scp進行傳輸
expdp導出,開並行,只導出元數據,排除JOB等存儲過程對象
scp伺服器相關許可權埠,賬戶密碼索取
impdp導入前,存儲空間,用戶許可權提前準備
二、操作記錄
--第一次導入操作failed
--失敗原因,導入表創建語法DDL失敗,原因
1.導入環境為XE學習環境,資料庫容量最大11g,如果空間足夠,不會報錯
2.導入建表語法,dump文件小於200M,導入建表語句還未插入數據,已占用11G存儲,由於表的STORAGE(INITIAL 65536 NEXT 1048576)參數導致,對每個段都分配了空間,導致創建空表占用11g達到XE閥值報錯
3 .測試11g新特效,延遲段創建對數據泵導入的對象無效,此參照只能影響手工新建的對象
1)資料獲取 數據泵INCLUDE and EXCLUDE對象類型視圖 Export/Import DataPump Parameters INCLUDE and EXCLUDE - How to Load and Unload Specific Objects (Doc ID 341733.1) ~database_export_objects /schema_export_objects /table_export_objects 2)進行導出 SQL> create directory dump as '/home/oracle/tools'; expdp \'/ as sysdba\' directory=dump dumpfile=system%u.dmp logfile=system.log SCHEMAS=system EXCLUDE=SEQUENCE,TRIGGER,REF_CONSTRAINT,CONSTRAINT,PROCDEPOBJ CONTENT=METADATA_ONLY cluster=n parallel --導出元數據,加了並行,但實質只導出一個dump文件,說明導出元數據只能串列導出 --附上導出日誌,如果願意,尾碼的導出類型,均可過濾,保留表、索引即可,最小化原則處理需求 Starting "SYS"."SYS_EXPORT_SCHEMA_02": "/******** AS SYSDBA" directory=dump dumpfile=system%u.dmp logfile=system.log SCHEMAS=system EXCLUDE=SEQUENCE,TRIGGER,REF_CONSTRAINT,CONSTRAINT,PROCDEPOBJ CONTENT=METADATA_ONLY cluster=n parallel=2 Processing object type SCHEMA_EXPORT/USER Processing object type SCHEMA_EXPORT/SYSTEM_GRANT Processing object type SCHEMA_EXPORT/ROLE_GRANT Processing object type SCHEMA_EXPORT/DEFAULT_ROLE Processing object type SCHEMA_EXPORT/PRE_SCHEMA/PROCACT_SCHEMA Processing object type SCHEMA_EXPORT/SYNONYM/SYNONYM Processing object type SCHEMA_EXPORT/TYPE/TYPE_SPEC Processing object type SCHEMA_EXPORT/TABLE/TABLE Processing object type SCHEMA_EXPORT/TABLE/PRE_TABLE_ACTION Processing object type SCHEMA_EXPORT/TABLE/GRANT/OWNER_GRANT/OBJECT_GRANT Processing object type SCHEMA_EXPORT/TABLE/COMMENT Processing object type SCHEMA_EXPORT/PACKAGE/PACKAGE_SPEC Processing object type SCHEMA_EXPORT/FUNCTION/FUNCTION Processing object type SCHEMA_EXPORT/PROCEDURE/PROCEDURE Processing object type SCHEMA_EXPORT/PACKAGE/COMPILE_PACKAGE/PACKAGE_SPEC/ALTER_PACKAGE_SPEC Processing object type SCHEMA_EXPORT/FUNCTION/ALTER_FUNCTION Processing object type SCHEMA_EXPORT/PROCEDURE/ALTER_PROCEDURE Processing object type SCHEMA_EXPORT/TABLE/INDEX/INDEX Processing object type SCHEMA_EXPORT/TABLE/INDEX/STATISTICS/INDEX_STATISTICS Processing object type SCHEMA_EXPORT/VIEW/VIEW Processing object type SCHEMA_EXPORT/VIEW/GRANT/OWNER_GRANT/OBJECT_GRANT Processing object type SCHEMA_EXPORT/VIEW/COMMENT Processing object type SCHEMA_EXPORT/PACKAGE/PACKAGE_BODY Processing object type SCHEMA_EXPORT/TABLE/STATISTICS/TABLE_STATISTICS Processing object type SCHEMA_EXPORT/TABLE/POST_TABLE_ACTION Processing object type SCHEMA_EXPORT/POST_SCHEMA/PROCACT_SCHEMA Master table "SYS"."SYS_EXPORT_SCHEMA_02" successfully loaded/unloaded ****************************************************************************** Dump file set for SYS.SYS_EXPORT_SCHEMA_02 is: /home/oracle/tools/system01.dmp Job "SYS"."SYS_EXPORT_SCHEMA_02" successfully completed at Thu Nov 29 14:48:14 2018 elapsed 0 00:00 3)scp 省略 4) 導入操作 --docker 命令進行SQL*Plus docker run -d --name "oradb" -v /home/dmuser/my_oracle_data:/u01/app/oracle IP:10082/sath89/oracle-xe-11g --正常環境,登陸伺服器端,即可使用impdp工具,本次場景為docker 有些特殊 --創建表空間 --源端用戶所在表空間 SQL> select tablespace_name from dba_segments where owner='SYSTEM' group by tablespace_name; TABLESPACE_NAME ------------------------------ SYSAUX SYSTEM--目標端創建表空間 crate tablesapce SYSTEM datafile '/u01/app/oracle/oradata/XE/system.dbf' size 10m uniform size 128k autoextend on next 30m maxsize 10g; sqlplus as sysdba create directories abc as 'xxx'; --導入 impdp \'/ as sysdba\' dumpfile=PICCPROD%U.dmp directory=dump logfile=picc.log REMAP_TABLESPACE=LIFELOB_DATA:LIFEDATA_T_L,LIFEINDEX_T_L:LIFEDATA_T_L --導入報錯 ORA-01658: unable to create INITIAL extent for segment in tablespace SYSTEM --對system表空間進行擴容 --再次導入報錯 CREATE TABLE "PICCPROD"."T_POLICY_CHECK_ITEM" ("ITEM_ID" NUMBER(10,0) NOT NULL ENABLE, "CUSTOMER_ID" NUMBER(10,0) NOT NULL ENABLE, "CHECK_NUM" NUMBER(4,0) NOT NULL ENABLE, "NORMAL" CHAR(1 BYTE) DEFAULT 'Y' NOT NULL ENABLE, "NOTES" VARCHAR2(1000 BYTE), "CHARGE_FEE" NUMBER(10,2) DEFAULT 0 NOT NULL ENABLE) PCTFREE 10 PCTUSED 40 IN ORA-39171: Job is experiencing a resumable wait. ORA-12953: The request exceeds the maximum allowed database size of 11 GB --搜索MOS 提示,學習XE類 Express Edition (簡化版本,資料庫允許最大11g) ORA-12953: The request exceeds the maximum allowed database size of 11 GB (Doc ID 2414879.1) [Release 11.2 to 12.2] This is an Oracle Database XE environment, and Oracle Database XE has a limitation of 11 GB of user data 疑問?測試庫怎麼有11g這麼大? SQL> select round( sum(bytes)/1024/1024) m from dba_segments where owner not in('PICCPROD'); M ---------- 1708 --導入用戶後 >11g出發XE峰值,報錯 SQL> select sum(bytes)/1024/1024 from dba_segments; SUM(BYTES)/1024/1024 -------------------- 11706.8125 --什麼類型的對象占用空間 SQL> select sum(bytes/1024/1024) ,segment_type from dba_segments where owner='PICCPROD' group by owner,segment_type; SUM(BYTES/1024/1024) SEGMENT_TYPE -------------------- ------------------ 9999 TABLE ?dump文件大小182,導入後,表大小9G oracle@c46bf408bfad:~/dmpfile$ ls PICCPROD01.dmp picc.log oracle@c46bf408bfad:~/dmpfile$ du -sm PICCPROD01.dmp 182 PICCPROD01.dmpfile 查詢表數據: select segment_name,segment_type,bytes/1024/1024 from dba_segments where owner='PICCPROD' and segment_type='TABLE' and rownum=1; SEGMENT_NAME SEGMENT_TYPE BYTES/1024/1024 ------------------------------ ------------------ --------------- T_DIAGNOSIS_TYPE TABLE .5 select count(*) from PICCPROD.T_DIAGNOSIS_TYPE; COUNT(*) ---------- 0 查詢表的區的分佈 SQL> select tablespace_name,segment_name,EXTENT_ID,BLOCK_ID,BYTES/1024,BLOCKS from dba_extents where owner='PICCPROD' and segment_name='T_DIAGNOSIS_TYPE'; TABLESPACE_NAME SEGMENT_NAME EXTENT_ID BLOCK_ID BYTES/1024 BLOCKS --------------- -------------------- ---------- ---------- ---------- ---------- LIFEDATA_T_L T_DIAGNOSIS_TYPE 0 128 128 16 LIFEDATA_T_L T_DIAGNOSIS_TYPE 1 144 128 16 LIFEDATA_T_L T_DIAGNOSIS_TYPE 2 160 128 16 LIFEDATA_T_L T_DIAGNOSIS_TYPE 3 176 128 16 嘗試對錶move(本次不再次對數據文件resize 進行分析,如果需要resize減少數據文件,需要從數據文件max_block_id 進行move,隨後對數據文件進行resize) alter table PICCPROD.T_DIAGNOSIS_TYPE move tablespace LIFEDATA_T_L; * ERROR at line 1: ORA-12953: The request exceeds the maximum allowed database size of 11 GB *Move時,需要存在一倍對象空閑空間,此時空閑空間不足無法Move* *釋放空間,找出max最大對象 select segment_name,sum(blocks) from dba_extents where owner='PICCPROD' group by segment_name order by 2; SEGMENT_NAME SUM(BLOCKS) ------------------------------------------- T_POLICY_AUTO 221312 T_POLICY_PRODUCT 309120 drop table PICCPROD.T_POLICY_AUTO purge; drop table PICCPROD.T_POLICY_PRODUCT purge; alter table PICCPROD.T_DIAGNOSIS_TYPE move tablespace LIFEDATA_T_L; select tablespace_name,segment_name,EXTENT_ID,BLOCK_ID,BYTES/1024,BLOCKS from dba_extents where owner='PICCPROD' and segment_name='T_DIAGNOSIS_TYPE'; TABLESPACE_NAME SEGMENT_NAME EXTENT_ID BLOCK_ID BYTES/1024 BLOCKS -------------------- -------------------- ---------- ---------- ---------- ---------- LIFEDATA_T_L T_DIAGNOSIS_TYPE 0 34720 128 16 LIFEDATA_T_L T_DIAGNOSIS_TYPE 1 34736 128 16 LIFEDATA_T_L T_DIAGNOSIS_TYPE 2 34752 128 16 LIFEDATA_T_L T_DIAGNOSIS_TYPE 3 34768 128 16 *嘗試對數據文件進行resize* select file_id,bytes/1024/1024 from dba_data_files where tablespace_name='LIFEDATA_T_L'; FILE_ID BYTES/1024/1024 ---------- --------------- 5 10000 alter database datafile 5 resize 5000M; * ERROR at line 1: ORA-03297: file contains used data beyond requested RESIZE value --雖然數據為空,但是表對應的段,分配了存儲,占用了存儲 *truncate table SQL> truncate table PICCPROD.T_DIAGNOSIS_TYPE; Table truncated. SQL> select tablespace_name,segment_name,EXTENT_ID,BLOCK_ID,BYTES/1024,BLOCKS from dba_extents where owner='PICCPROD' and segment_name='T_DIAGNOSIS_TYPE'; TABLESPACE_NAME SEGMENT_NAME EXTENT_ID BLOCK_ID BYTES/1024 BLOCKS -------------------- -------------------- ---------- ---------- ---------- ---------- LIFEDATA_T_L T_DIAGNOSIS_TYPE 0 34720 128 16 LIFEDATA_T_L T_DIAGNOSIS_TYPE 1 34736 128 16 LIFEDATA_T_L T_DIAGNOSIS_TYPE 2 34752 128 16 LIFEDATA_T_L T_DIAGNOSIS_TYPE 3 34768 128 16 *資料庫參數,延遲段創建 SQL> show parameter defer NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ deferred_segment_creation boolean TRUE --參數並未關閉--false,但是之前的導入還是創建表,因此延遲段創建參數,對於數據泵導入的對象是不起作用的 *查詢建表語句 --導入 impdp \'/ as sysdba\' dumpfile=PICCPROD%U.dmp directory=dump logfile=picc%U.log sqlfile=sql01.sql ORA-39002: invalid operation ORA-31694: master table "SYS"."SYS_SQL_FILE_FULL_01" failed to load/unload ORA-02354: error in exporting/importing data ORA-39776: fatal Direct Path API error loading table "SYS"."SYS_SQL_FILE_FULL_01" ORA-12953: The request exceeds the maximum allowed database size of 11 GB -- CREATE TABLE "PICCPROD"."T_POLICY_CHECK_ITEM" ("ITEM_ID" NUMBER(10,0) NOT NULL ENABLE, "CUSTOMER_ID" NUMBER(10,0) NOT NULL ENABLE, "CHECK_NUM" NUMBER(4,0) NOT NULL ENABLE, "NORMAL" CHAR(1 BYTE) DEFAULT 'Y' NOT NULL ENABLE, "NOTES" VARCHAR2(1000 BYTE), "CHARGE_FEE" NUMBER(10,2) DEFAULT 0 NOT NULL ENABLE) PCTFREE 10 PCTUSED 40 IN ORA-39171: Job is experiencing a resumable wait. ORA-12953: The request exceeds the maximum allowed database size of 11 GB 為何impdp,提取ddl語句也報錯,因為數據泵導入會在資料庫內,創建相關對象,占用臨時存儲,資料庫都滿了,因此即使不導入數據,臨時創建個對象都是報錯的
--第二次操作,successful
總結:第一次導入失敗,主要原因是,數據泵導入後,分配段初始化空間,避免導入分配第一個extent,即避免現象
impdp \'/ as sysdba\' dumpfile=PICCPROD%U.dmp directory=dump logfile=picc.log REMAP_TABLESPACE=LIFELOB_DATA:LIFEDATA_T_L,LIFEINDEX_T_L:LIFEDATA_T_L
TRANSFORM = SEGMENT_ATTRIBUTES:N
三、問題反思
TRANSFORM = SEGMENT_ATTRIBUTES:N 參數作用何在,有什麼特性 實驗測試: 1.創建一個測試表a, 創建主鍵,創建一個索引 2.對對象進行元數據導出 3.實驗參數,進行導入 4.查詢及驗證表及索引是否存在 5.查詢表及索引的段是否分配 6.impdp sqlfile 查詢與表 and 索引 get ddl 差異
1.創建一個測試表a, 創建主鍵,創建一個索引 SQL> create table yang.ceshi as select * from scott.emp; SQL> alter table yang.ceshi add constraint pk_id primary key(empno); SQL> create index yang.id2 on yang.ceshi(deptno); 2.對對象進行元數據導出 expdp \'/ as sysdba\' directory=dump dumpfile=yang%u.dmp logfile=yang.log SCHEMAS=yang include=TABLE:"IN('CESHI')" CONTENT=METADATA_ONLY 3.實驗參數,進行導入 impdp \'/ as sysdba\' directory=dump dumpfile=yang%u.dmp logfile=yang1.log TRANSFORM = SEGMENT_ATTRIBUTES:N REMAP_SCHEMA=yang:scott 4.查詢及驗證表及索引是否存在 select object_name,object_type,status from user_objects where created>sysdate-1 OBJECT_NAME OBJECT_TYPE STATUS -------------------- ------------------- ------- ID2 INDEX VALID PK_ID INDEX VALID CESHI TABLE VALID 5.查詢表及索引的段是否分配 SQL> select SEGMENT_NAME,SEGMENT_TYPE,BLOCKS,EXTENTS,INITIAL_EXTENT,NEXT_EXTENT from user_segments where segment_name in(select object_name from user_objects where created>sysdate-1); no rows selected 6.impdp sqlfile 查詢與表 and 索引 get ddl 差異 [oracle@t2 tools]$ impdp \'/ as sysdba\' directory=dump dumpfile=yang%u.dmp logfile=yang1.log TRANSFORM = SEGMENT_ATTRIBUTES:N REMAP_SCHEMA=yang:scott sqlfile=test_table.sql --查詢創建語法,進行說明: 第一部分 alter session level trace文件 第二部分,創建表語法,沒有了storage語法 第三部分,創建索引,對索引取消並行,在真實導入數據過程中,parallel並行能幫助快速創建索引,創建索引後數據泵自動noparallel 第四部分,導入完畢後,對數據泵臨時創建對象進行清理 [oracle@t2 tools]$ more test_table.sql -- CONNECT SYS ALTER SESSION SET EVENTS '22830 TRACE NAME CONTEXT FOREVER, LEVEL 192 '; -- new object type path: SCHEMA_EXPORT/TABLE/TABLE CREATE TABLE "SCOTT"."CESHI" ( "EMPNO" NUMBER(4,0), "ENAME" VARCHAR2(10 BYTE), "JOB" VARCHAR2(9 BYTE), "MGR" NUMBER(4,0), "HIREDATE" DATE, "SAL" NUMBER(7,2), "COMM" NUMBER(7,2), "DEPTNO" NUMBER(2,0) ) ; -- new object type path: SCHEMA_EXPORT/TABLE/INDEX/INDEX -- CONNECT SCOTT CREATE UNIQUE INDEX "SCOTT"."PK_ID" ON "SCOTT"."CESHI" ("EMPNO") ; ALTER INDEX "SCOTT"."PK_ID" NOPARALLEL; CREATE INDEX "SCOTT"."ID2" ON "SCOTT"."CESHI" ("DEPTNO") ; ALTER INDEX "SCOTT"."ID2" NOPARALLEL; -- new object type path: SCHEMA_EXPORT/TABLE/CONSTRAINT/CONSTRAINT -- CONNECT SYS ALTER TABLE "SCOTT"."CESHI" ADD CONSTRAINT "PK_ID" PRIMARY KEY ("EMPNO") USING INDEX "SCOTT"."PK_ID" ENABLE; -- new object type path: SCHEMA_EXPORT/TABLE/INDEX/STATISTICS/INDEX_STATISTICS DECLARE I_N VARCHAR2(60); I_O VARCHAR2(60); NV VARCHAR2(1); c DBMS_METADATA.T_VAR_COLL; df varchar2(21) := 'YYYY-MM-DD:HH24:MI:SS'; stmt varchar2(300) := ' INSERT INTO "SYS"."IMPDP_STATS" (type,version,flags,c1,c2,c3,c5,n1,n2,n3,n4,n5,n6,n7,n8,n9,n10,n11,n12,d1,cl1) VALUES (''I'',6,:1,: 2,:3,:4,:5,:6,:7,:8,:9,:10,:11,:12,:13,NULL,:14,:15,NULL,:16,:17)'; BEGIN DELETE FROM "SYS"."IMPDP_STATS"; i_n := 'PK_ID'; i_o := 'SCOTT'; EXECUTE IMMEDIATE stmt USING 2,I_N,NV,NV,I_O,14,1,14,1,1,1,0,14,NV,NV,TO_DATE('2018-11-29 15:17:22',df),NV; DBMS_STATS.IMPORT_INDEX_STATS('"' || i_o || '"','"' || i_n || '"',NULL,'"IMPDP_STATS"',NULL,'"SYS"'); DELETE FROM "SYS"."IMPDP_STATS"; END; / DECLARE I_N VARCHAR2(60); I_O VARCHAR2(60); NV VARCHAR2(1); c DBMS_METADATA.T_VAR_COLL; df varchar2(21) := 'YYYY-MM-DD:HH24:MI:SS'; stmt varchar2(300) := ' INSERT INTO "SYS"."IMPDP_STATS" (type,version,flags,c1,c2,c3,c5,n1,n2,n3,n4,n5,n6,n7,n8,n9,n10,n11,n12,d1,cl1) VALUES (''I'',6,:1,: 2,:3,:4,:5,:6,:7,:8,:9,:10,:11,:12,:13,NULL,:14,:15,NULL,:16,:17)'; BEGIN DELETE FROM "SYS"."IMPDP_STATS"; i_n := 'ID2'; i_o := 'SCOTT'; EXECUTE IMMEDIATE stmt USING 2,I_N,NV,NV,I_O,14,1,3,1,1,1,0,14,NV,NV,TO_DATE('2018-11-29 15:18:08',df),NV; DBMS_STATS.IMPORT_INDEX_STATS('"' || i_o || '"','"' || i_n || '"',NULL,'"IMPDP_STATS"',NULL,'"SYS"'); DELETE FROM "SYS"."IMPDP_STATS"; END; / ---查詢表的ddl語法 select dbms_metadata.get_ddl('TABLE','CESHI','SCOTT') ddl_text from dual DDL_TEXT -------------------------------------------------------------------------------- CREATE TABLE "SCOTT"."CESHI" ( "EMPNO" NUMBER(4,0), "ENAME" VARCHAR2(10), "JOB" VARCHAR2(9), "MGR" NUMBER(4,0), "HIREDATE" DATE, "SAL" NUMBER(7,2), "COMM" NUMBER(7,2), "DEPTNO" NUMBER(2,0), CONSTRAINT "PK_ID" PRIMARY KEY ("EMPNO") USING INDEX PCTFREE 10 INITRANS 2 MAXTRANS 255 COMPUTE STATISTICS TABLESPACE "USERS" ENABLE ) SEGMENT CREATION DEFERRED PCTFREE 10 PCTUSED 40 INITRANS 1 MAXTRANS 255 NOCOMPRESS LOGGING TABLESPACE "USERS" --表不存在storage屬性後 insert 數據進行測試 SQL> insert into scott.ceshi select * from scott.emp; --查詢表及索引的extent分配,發現表其實並沒有extent storage屬性,是怎麼分配的呢? SQL> select SEGMENT_NAME,SEGMENT_TYPE,BLOCKS,EXTENTS,INITIAL_EXTENT/1024 INITIAL_EXTENT_K,NEXT_EXTENT/1024 NEXT_EXTENT_K from user_segments where segment_name in(select object_name from user_objects where created>sysdate-1); SEGMENT_NAME SEGMENT_TYPE BLOCKS EXTENTS INITIAL_EXTENT_K NEXT_EXTENT_K --------------- ------------------ ---------- ---------- ---------------- ------------- CESHI TABLE 8 1 64 1024 PK_ID INDEX 8 1 64 1024 ID2 INDEX 8 1 64 1024 --查詢表空間屬性 SQL> select tablespace_name from user_segments where segment_name in(select object_name from user_objects where created>sysdate-1) group by tablespace_name; TABLESPACE_NAME ------------------------------ USERS SQL> select TABLESPACE_NAME,BLOCK_SIZE,INITIAL_EXTENT/1024 INITIAL_EXTENT_K,NEXT_EXTENT/1024 NEXT_EXTENT_K,SEGMENT_SPACE_MANAGEMENT,ALLOCATION_TYPE,EXTENT_MANAGEMENT from dba_tablespaces where tablespace_name='USERS'; TABLESPACE BLOCK_SIZE INITIAL_EXTENT_K NEXT_EXTENT_K SEGMEN ALLOCATIO EXTENT_MAN ---------- ---------- ---------------- ------------- ------ --------- ---------- USERS 8192 64 AUTO SYSTEM LOCAL --表空間本地管理表空間,系統自動擴展分配, 8K,1M,8M,64M --可見,對於表來說,如果未配置storage屬性,則表使用表空間的預設storage屬性分配extents 大小
四、問題總結
1.根據問題現象,尋找解決方法,路不止一條,在能接受的範圍內解決即可
2.根據問題現象,尋找合適的技術點,使用自己能運用且無風險的方案解決問題,測試後方可使用