本文分享自華為雲社區《你的JoinHint為什麼不生效【綻放吧!GaussDB(DWS)雲原生數倉】》,作者:你是猴子請來的救兵嗎 。 引言 提起資料庫的Hint,幾乎每個DBA都知道這一強大功能。在GaussDB(DWS)中,Hint可以被用來干預SQL的執行計劃,但是在日常工作中,很多開發人員對 ...
本文分享自華為雲社區《你的JoinHint為什麼不生效【綻放吧!GaussDB(DWS)雲原生數倉】》,作者:你是猴子請來的救兵嗎 。
引言
提起資料庫的Hint,幾乎每個DBA都知道這一強大功能。在GaussDB(DWS)中,Hint可以被用來干預SQL的執行計劃,但是在日常工作中,很多開發人員對Hint的缺乏深入瞭解,經常遇到Hint失效的情況卻又束手無策。
本次針對JoinHint從案例著手深入解析JoinHint不生效的原因,以便讀者能“知其所以然”。(本文不討論Hint的基礎語法問題)。
問題案例
內核版本 GaussDB 8.1.3
問題描述 兩表關聯查詢,使用hashjoin hint干預join方式,但hint不生效
問題用例
CREATE TABLE workitem ( language character varying(10), userid character varying(240), opiontype character varying(240), processinstid character varying(240), workitemid character varying(240), type_name character varying(240), type_code character varying(240), createtime timestamp without time zone, endtime timestamp without time zone, notrejecttotal numeric, dws_created_time timestamp without time zone ) WITH (orientation=column, compression=low, colversion=2.0, enable_delta=false) DISTRIBUTE BY HASH(workitemid); CREATE TABLE workday ( mm timestamp with time zone, rn numeric ) WITH (orientation=column, compression=low, colversion=2.0, enable_delta=false) DISTRIBUTE BY HASH(mm); explain SELECT /*+ hashjoin(c d) */ c.userid,c.type_name,c.type_code,count(1) num FROM workitem c INNER JOIN workday d ON c.createtime = d.mm WHERE c.createtime >= '2023-09-01' AND c.endtime < '2023-10-01' GROUP BY c.userid,c.type_name,c.type_code; WARNING: unused hint: HashJoin(c d) QUERY PLAN ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- id | operation | E-rows | E-memory | E-width | E-costs ----+------------------------------------------------------+--------+----------+---------+--------- 1 | -> Row Adapter | 2 | | 1502 | 33.12 2 | -> Vector Sonic Hash Aggregate | 2 | | 1502 | 33.12 3 | -> Vector Streaming (type: GATHER) | 4 | | 1502 | 33.12 4 | -> Vector Sonic Hash Aggregate | 4 | 16MB | 1502 | 27.12 5 | -> Vector Nest Loop (6,8) | 5 | 1MB | 1494 | 27.08 6 | -> Vector Streaming(type: BROADCAST) | 14 | 2MB | 8 | 13.68 7 | -> CStore Scan on workday d | 7 | 1MB | 8 | 13.05 8 | -> Vector Materialize | 5 | 16MB | 1502 | 13.09 9 | -> CStore Scan on workitem c | 5 | 1MB | 1502 | 13.08 RunTime Analyze Information ------------------------------------------- "public.workitem" runtime: 25.794ms "public.workday" runtime: 18.098ms Predicate Information (identified by plan id) -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 5 --Vector Nest Loop (6,8) Join Filter: (c.createtime = d.mm) 7 --CStore Scan on workday d Filter: (mm >= '2023-09-01 00:00:00'::timestamp without time zone) Pushdown Predicate Filter: (mm >= '2023-09-01 00:00:00'::timestamp without time zone) 9 --CStore Scan on workitem c Filter: ((createtime >= '2023-09-01 00:00:00'::timestamp without time zone) AND (endtime < '2023-10-01 00:00:00'::timestamp without time zone)) Pushdown Predicate Filter: ((createtime >= '2023-09-01 00:00:00'::timestamp without time zone) AND (endtime < '2023-10-01 00:00:00'::timestamp without time zone)) ====== Query Summary ===== ------------------------------- System available mem: 4710400KB Query Max mem: 4710400KB Query estimated mem: 5271KB (33 rows)
問題定位
嘗試關閉nestloop路徑,來驗證是否可以生成hash計劃
set enable_nestloop = off;
set enable_mergejoin = off;
set enable_hashjoin = on;
WARNING: unused hint: HashJoin(c d) QUERY PLAN ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- id | operation | E-rows | E-memory | E-width | E-costs ----+------------------------------------------------------+--------+----------+---------+--------- 1 | -> Row Adapter | 2 | | 1502 | 33.12 2 | -> Vector Sonic Hash Aggregate | 2 | | 1502 | 33.12 3 | -> Vector Streaming (type: GATHER) | 4 | | 1502 | 33.12 4 | -> Vector Sonic Hash Aggregate | 4 | 16MB | 1502 | 27.12 5 | -> Vector Nest Loop (6,8) | 5 | 1MB | 1494 | 27.08 6 | -> Vector Streaming(type: BROADCAST) | 14 | 2MB | 8 | 13.68 7 | -> CStore Scan on workday d | 7 | 1MB | 8 | 13.05 8 | -> Vector Materialize | 5 | 16MB | 1502 | 13.09 9 | -> CStore Scan on workitem c | 5 | 1MB | 1502 | 13.08 Predicate Information (identified by plan id) -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 5 --Vector Nest Loop (6,8) Join Filter: (c.createtime = d.mm) 7 --CStore Scan on workday d Filter: (mm >= '2023-09-01 00:00:00'::timestamp without time zone) Pushdown Predicate Filter: (mm >= '2023-09-01 00:00:00'::timestamp without time zone) 9 --CStore Scan on workitem c Filter: ((createtime >= '2023-09-01 00:00:00'::timestamp without time zone) AND (endtime < '2023-10-01 00:00:00'::timestamp without time zone)) Pushdown Predicate Filter: ((createtime >= '2023-09-01 00:00:00'::timestamp without time zone) AND (endtime < '2023-10-01 00:00:00'::timestamp without time zone)) ====== Query Summary ===== ------------------------------- System available mem: 4710400KB Query Max mem: 4710400KB Query estimated mem: 5271KB (28 rows)
關閉nestloop路徑後,仍然生成nestloop計劃,且E-costs代價中未添加懲罰代價,說明該場景語句本身不支持hashjoin。
檢查關聯表達式(c.createtime = d.mm),確認是否支持hashjoin。
- 關聯表達式為欄位關聯,不存在函數嵌套
- 關聯表達式兩邊數據類型為timestamp without time zone和timestamp with time zone,通過系統表pg_operator確認是否支持hashjoin。
postgres=# select * from pg_operator where oprname = '=' and oprleft = 'timestamp'::regtype and oprright = 'timestamptz'::regtype; -[ RECORD 1 ]+------------------------- oprname | = oprnamespace | 11 oprowner | 10 oprkind | b oprcanmerge | t oprcanhash | f oprleft | 1114 oprright | 1184 oprresult | 16 oprcom | 2542 oprnegate | 2539 oprcode | timestamp_eq_timestamptz oprrest | eqsel oprjoin | eqjoinsel
- 通過結果確認oprcanhash為false,代表該操作符不支持hash連接;原因是,左邊數據不帶時區,右邊數據帶,在比較時要先處理時區問題,不能直接拿存儲值進行判斷。
改善辦法
通過系統表確認timestamp類型的等值關聯和timestamptz的等值關聯均支持hash連接。postgres=# select * from pg_operator where oprname = '=' and oprleft = oprright and oprleft in('timestamp'::regtype,'timestamptz'::regtype); -[ RECORD 1 ]+--------------- oprname | = oprnamespace | 11 oprowner | 10 oprkind | b oprcanmerge | t oprcanhash | t oprleft | 1184 oprright | 1184 oprresult | 16 oprcom | 1320 oprnegate | 1321 oprcode | timestamptz_eq oprrest | eqsel oprjoin | eqjoinsel -[ RECORD 2 ]+--------------- oprname | = oprnamespace | 11 oprowner | 10 oprkind | b oprcanmerge | t oprcanhash | t oprleft | 1114 oprright | 1114 oprresult | 16 oprcom | 2060 oprnegate | 2061 oprcode | timestamp_eq oprrest | eqsel oprjoin | eqjoinsel
在關聯條件上添加類型轉換,保證兩邊類型一致,即(c.createtime::timestamptz = d.mm)或(c.createtime = d.mm::timestamp)。
postgres=# explain postgres-# SELECT /*+ hashjoin(c d) */ postgres-# c.userid,c.type_name,c.type_code,count(1) num postgres-# FROM workitem c INNER JOIN workday d ON c.createtime::timestamptz = d.mm postgres-# WHERE c.createtime >= '2023-09-01' AND c.endtime < '2023-10-01' postgres-# GROUP BY c.userid,c.type_name,c.type_code; QUERY PLAN ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- id | operation | E-rows | E-memory | E-width | E-costs ----+------------------------------------------------------+--------+----------+---------+--------- 1 | -> Row Adapter | 2 | | 1502 | 34.29 2 | -> Vector Sonic Hash Aggregate | 2 | | 1502 | 34.29 3 | -> Vector Streaming (type: GATHER) | 4 | | 1502 | 34.29 4 | -> Vector Sonic Hash Aggregate | 4 | 16MB | 1502 | 28.29 5 | -> Vector Sonic Hash Join (6,8) | 5 | 16MB | 1494 | 28.25 6 | -> Vector Streaming(type: BROADCAST) | 40 | 2MB | 8 | 15.06 7 | -> CStore Scan on workday d | 20 | 1MB | 8 | 13.01 8 | -> CStore Scan on workitem c | 5 | 1MB | 1502 | 13.08 Predicate Information (identified by plan id) -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 5 --Vector Sonic Hash Join (6,8) Hash Cond: (d.mm = (c.createtime)::timestamp with time zone) 8 --CStore Scan on workitem c Filter: ((createtime >= '2023-09-01 00:00:00'::timestamp without time zone) AND (endtime < '2023-10-01 00:00:00'::timestamp without time zone)) Pushdown Predicate Filter: ((createtime >= '2023-09-01 00:00:00'::timestamp without time zone) AND (endtime < '2023-10-01 00:00:00'::timestamp without time zone)) ====== Query Summary ===== ------------------------------- System available mem: 4710400KB Query Max mem: 4710400KB Query estimated mem: 5530KB (24 rows) postgres=# explain SELECT /*+ hashjoin(c d) */ c.userid,c.type_name,c.type_code,count(1) num FROM workitem c INNER JOIN workday d ON c.createtime = d.mm::timestamp WHERE c.createtime >= '2023-09-01' AND c.endtime < '2023-10-01' GROUP BY c.userid,c.type_name,c.type_code; QUERY PLAN ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- id | operation | E-rows | E-memory | E-width | E-costs ----+------------------------------------------------------+--------+----------+---------+--------- 1 | -> Row Adapter | 2 | | 1502 | 32.91 2 | -> Vector Sonic Hash Aggregate | 2 | | 1502 | 32.91 3 | -> Vector Streaming (type: GATHER) | 4 | | 1502 | 32.91 4 | -> Vector Sonic Hash Aggregate | 4 | 16MB | 1502 | 26.91 5 | -> Vector Sonic Hash Join (6,8) | 5 | 16MB | 1494 | 26.87 6 | -> Vector Streaming(type: BROADCAST) | 14 | 2MB | 8 | 13.71 7 | -> CStore Scan on workday d | 7 | 1MB | 8 | 13.08 8 | -> CStore Scan on workitem c | 5 | 1MB | 1502 | 13.08 Predicate Information (identified by plan id) -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 5 --Vector Sonic Hash Join (6,8) Hash Cond: ((d.mm)::timestamp without time zone = c.createtime) 7 --CStore Scan on workday d Filter: ((mm)::timestamp without time zone >= '2023-09-01 00:00:00'::timestamp without time zone) 8 --CStore Scan on workitem c Filter: ((createtime >= '2023-09-01 00:00:00'::timestamp without time zone) AND (endtime < '2023-10-01 00:00:00'::timestamp without time zone)) Pushdown Predicate Filter: ((createtime >= '2023-09-01 00:00:00'::timestamp without time zone) AND (endtime < '2023-10-01 00:00:00'::timestamp without time zone)) ====== Query Summary ===== ------------------------------- System available mem: 4710400KB Query Max mem: 4710400KB Query estimated mem: 5530KB (26 rows)
知識小結
實際使用過程中導致hint生效的原因很多,這裡總結排查hashjoin hint步驟以供參考:
- 檢查hint中的表名是否正確、是否存在重名、是否在當前層可見,此類場景通常在explain中會給出提示,自行排查即可。
- 判斷關聯hint中的表名是否被提升導致表名不存在,此類場景通常在explain中會給出提示
postgres=# explain SELECT /*+ hashjoin(c d) */ c.userid,c.type_name,c.type_code,count(1) num FROM workitem c INNER JOIN (select * from workday where mm >= '2023-09-01') d ON c.createtime = d.mm::timestamp WHERE c.createtime >= '2023-09-01' AND c.endtime < '2023-10-01' GROUP BY c.userid,c.type_name,c.type_code; WARNING: Error hint: HashJoin(c d), relation name "d" is not found. QUERY PLAN ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- id | operation | E-rows | E-memory | E-width | E-costs ----+------------------------------------------------------+--------+----------+---------+--------- 1 | -> Row Adapter | 2 | | 1502 | 32.78 2 | -> Vector Sonic Hash Aggregate | 2 | | 1502 | 32.78 3 | -> Vector Streaming (type: GATHER) | 4 | | 1502 | 32.78 4 | -> Vector Sonic Hash Aggregate | 4 | 16MB | 1502 | 26.78 5 | -> Vector Sonic Hash Join (6,8) | 5 | 16MB | 1494 | 26.74 6 | -> Vector Streaming(type: BROADCAST) | 10 | 2MB | 8 | 13.58 7 | -> CStore Scan on workday | 5 | 1MB | 8 | 13.11 8 | -> CStore Scan on workitem c | 5 | 1MB | 1502 | 13.08 Predicate Information (identified by plan id) -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 5 --Vector Sonic Hash Join (6,8) Hash Cond: ((workday.mm)::timestamp without time zone = c.createtime) 7 --CStore Scan on workday Filter: ((mm >= '2023-09-01 00:00:00+08'::timestamp with time zone) AND ((mm)::timestamp without time zone >= '2023-09-01 00:00:00'::timestamp without time zone)) Pushdown Predicate Filter: (mm >= '2023-09-01 00:00:00+08'::timestamp with time zone) 8 --CStore Scan on workitem c Filter: ((createtime >= '2023-09-01 00:00:00'::timestamp without time zone) AND (endtime < '2023-10-01 00:00:00'::timestamp without time zone)) Pushdown Predicate Filter: ((createtime >= '2023-09-01 00:00:00'::timestamp without time zone) AND (endtime < '2023-10-01 00:00:00'::timestamp without time zone)) ====== Query Summary ===== ------------------------------- System available mem: 4710400KB Query Max mem: 4710400KB Query estimated mem: 5530KB (27 rows)
針對此種情況,8.2.0及以上版本可以通過添加no merge hint來禁用子查詢提升從而規避hint失效問題。
通過join路徑參數驗證目標路徑是否可生效。--如通過關閉其他路徑參數來驗證某一路徑是否可達 set enable_nestloop = off; set enable_mergejoin = off; set enable_hashjoin = on;檢查關聯條件中是否存在volatile函數。
postgres=# create or replace function gettimediff(timestamp) returns interval language sql as 'select $1-timeofday()::timestamp' volatile; CREATE FUNCTION postgres=# explain SELECT /*+ hashjoin(c d) */ c.userid,c.type_name,c.type_code,count(1) num FROM workitem c INNER JOIN workday d ON gettimediff(c.createtime) = gettimediff(d.mm::timestamp) WHERE c.createtime >= '2023-09-01' AND c.endtime < '2023-10-01' GROUP BY c.userid,c.type_name,c.type_code; WARNING: unused hint: HashJoin(c d) QUERY PLAN ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- id | operation | E-rows | E-width | E-costs ----+-------------------------------------------------------------+--------+---------+--------- 1 | -> HashAggregate | 5 | 1502 | 3.10 2 | -> Nested Loop (3,4) | 5 | 1494 | 3.00 3 | -> Data Node Scan on workitem "_REMOTE_TABLE_QUERY_" | 5 | 1502 | 0.00 4 | -> Data Node Scan on workday "_REMOTE_TABLE_QUERY_" | 20 | 8 | 0.00 Predicate Information (identified by plan id) ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2 --Nested Loop (3,4) Join Filter: ((c.createtime - (timeofday())::timestamp without time zone) = ((d.mm)::timestamp without time zone - (timeofday())::timestamp without time zone)) (11 rows)檢查關聯條件中兩表欄位是否在等號兩側,若不是則進行調整。
postgres=# explain SELECT /*+ hashjoin(c d) */ c.userid,c.type_name,c.type_code,count(1) num FROM workitem c INNER JOIN workday d ON ifnull(c.createtime,d.mm) = now() WHERE c.createtime >= '2023-09-01' AND c.endtime < '2023-10-01' GROUP BY c.userid,c.type_name,c.type_code; WARNING: unused hint: HashJoin(c d) QUERY PLAN ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- id | operation | E-rows | E-memory | E-width | E-costs ----+------------------------------------------------------+--------+----------+---------+--------- 1 | -> Row Adapter | 1 | | 1502 | 35.37 2 | -> Vector Sonic Hash Aggregate | 1 | | 1502 | 35.37 3 | -> Vector Streaming (type: GATHER) | 2 | | 1502 | 35.37 4 | -> Vector Sonic Hash Aggregate | 2 | 16MB | 1502 | 29.37 5 | -> Vector Nest Loop (6,8) | 2 | 1MB | 1494 | 29.35 6 | -> Vector Streaming(type: BROADCAST) | 40 | 2MB | 8 | 15.06 7 | -> CStore Scan on workday d | 20 | 1MB | 8 | 13.01 8 | -> Vector Materialize | 5 | 16MB | 1502 | 13.09 9 | -> CStore Scan on workitem c | 5 | 1MB | 1502 | 13.08 Predicate Information (identified by plan id) -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 5 --Vector Nest Loop (6,8) Join Filter: (COALESCE((c.createtime)::timestamp with time zone, d.mm) = now()) 9 --CStore Scan on workitem c Filter: ((createtime >= '2023-09-01 00:00:00'::timestamp without time zone) AND (endtime < '2023-10-01 00:00:00'::timestamp without time zone)) Pushdown Predicate Filter: ((createtime >= '2023-09-01 00:00:00'::timestamp without time zone) AND (endtime < '2023-10-01 00:00:00'::timestamp without time zone)) ====== Query Summary ===== ------------------------------- System available mem: 4710400KB Query Max mem: 4710400KB Query estimated mem: 5275KB檢查關聯條件是否為等值關聯,若不是則進行調整。
(25 rows)
postgres=# explain SELECT /*+ hashjoin(c d) */ c.userid,c.type_name,c.type_code,count(1) num FROM workitem c INNER JOIN workday d ON c.createtime::timestamptz > d.mm WHERE c.createtime >= '2023-09-01' AND c.endtime < '2023-10-01' GROUP BY c.userid,c.type_name,c.type_code; WARNING: unused hint: HashJoin(c d) QUERY PLAN ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- id | operation | E-rows | E-memory | E-width | E-costs ----+------------------------------------------------------+--------+----------+---------+--------- 1 | -> Row Adapter | 5 | | 1502 | 35.41 2 | -> Vector Sonic Hash Aggregate | 5 | | 1502 | 35.41 3 | -> Vector Streaming (type: GATHER) | 10 | | 1502 | 35.41 4 | -> Vector Sonic Hash Aggregate | 10 | 16MB | 1502 | 29.41 5 | -> Vector Nest Loop (6,8) | 33 | 1MB | 1494 | 29.20 6 | -> Vector Streaming(type: BROADCAST) | 40 | 2MB | 8 | 15.06 7 | -> CStore Scan on workday d | 20 | 1MB | 8 | 13.01 8 | -> Vector Materialize | 5 | 16MB | 1502 | 13.09 9 | -> CStore Scan on workitem c | 5 | 1MB | 1502 | 13.08 Predicate Information (identified by plan id) -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 5 --Vector Nest Loop (6,8) Join Filter: ((c.createtime)::timestamp with time zone > d.mm) 9 --CStore Scan on workitem c Filter: ((createtime >= '2023-09-01 00:00:00'::timestamp without time zone) AND (endtime < '2023-10-01 00:00:00'::timestamp without time zone)) Pushdown Predicate Filter: ((createtime >= '2023-09-01 00:00:00'::timestamp without time zone) AND (endtime < '2023-10-01 00:00:00'::timestamp without time zone)) ====== Query Summary ===== ------------------------------- System available mem: 4710400KB Query Max mem: 4710400KB Query estimated mem: 5281KB (25 rows)檢查關聯關係兩側的數據類型,並通過pg_operator.oprcanhash確認是否支持hash連接,若不支持則需改寫為支持的操作符。
postgres=# select * from pg_operator where oprname = '=' and oprleft = 'timestamp'::regtype and oprright = 'timestamptz'::regtype; -[ RECORD 1 ]+------------------------- oprname | = oprnamespace | 11 oprowner | 10 oprkind | b oprcanmerge | t oprcanhash | f oprleft | 1114 oprright | 1184 oprresult | 16 oprcom | 2542 oprnegate | 2539 oprcode | timestamp_eq_timestamptz oprrest | eqsel oprjoin | eqjoinsel如果是指定join順序的hint,如leading(c e),會存在邏輯本身衝突導致hint失敗的情況。
postgres=# explain SELECT /*+ leading(c e) */ c.userid,c.type_name,c.type_code,count(1) num FROM workitem c LEFT JOIN workday d ON c.createtime = d.mm LEFT JOIN workday e ON d.mm = e.mm WHERE c.createtime >= '2023-09-01' AND c.endtime < '2023-10-01' GROUP BY c.userid,c.type_name,c.type_code; WARNING: unused hint: Leading(c e)