一、前言

2018年寫過一篇分庫分表的文章《SpringBoot使用sharding-jdbc分庫分表》，但是存在很多不完美的地方比如：

• sharding-jdbc的版本（1.4.2）過低，現在github上的最新版本都是5.3.2了，很多用法和API都過時了。
• 分庫分表配置採用Java硬編碼的方式不夠靈活
• 持久層使用的是spring-boot-starter-data-jpa，而不是主流的mybatis+mybatis-plus+druid-spring-boot-stater
• 沒有支持自定義主鍵生成策略

二、設計思路

針對上述問題，本人計劃開發一個通用的分庫分表starter，具備以下特性：

1. 基於ShardingSphere-JDBC版本4.1.1，官方支持的特性我們都支持
2. 支持yaml文件配置，無需編碼開箱即用
3. 支持多種數據源，整合主流的mybatis
4. 支持自定義主鍵生成策略，並提供預設的雪花演算法實現

通過查看官方文檔，可以發現starter的核心邏輯就是獲取分庫分表等配置，然後在自動配置類創建數據源註入Spring容器即可。

三、編碼實現

3.1 starter工程搭建

首先創建一個spring-boot-starter工程ship-sharding-spring-boot-starter，不會的小伙伴可以參考以前寫的教程《【SpringBoot】編寫一個自己的Starter》。

創建自動配置類cn.sp.sharding.config.ShardingAutoConfig，併在resources/META-INF/spring.factories文件中配置自動配置類的全路徑。

org.springframework.boot.autoconfigure.EnableAutoConfiguration=cn.sp.sharding.config.ShardingAutoConfig

然後需要在pom.xml文件引入sharding-jbc依賴和工具包guava。

    <properties>
        <java.version>8</java.version>
        <spring-boot.version>2.4.0</spring-boot.version>
        <sharding-jdbc.version>4.1.1</sharding-jdbc.version>
    </properties>
    
      <dependency>
            <groupId>org.apache.shardingsphere</groupId>
            <artifactId>sharding-jdbc-core</artifactId>
            <version>${sharding-jdbc.version}</version>
        </dependency>

        <dependency>
            <groupId>com.google.guava</groupId>
            <artifactId>guava</artifactId>
            <version>18.0</version>
        </dependency>

3.2 註入ShardingDataSource

分庫分表配置這塊，為了方便自定義配置首碼，創建ShardingRuleConfigurationProperties類繼承sharding-jbc的YamlShardingRuleConfiguration類即可，代碼如下：

/**
 * @author Ship
 * @version 1.0.0
 * @description:
 * @date 2023/06/06
 */
@ConfigurationProperties(prefix = CommonConstants.COMMON_CONFIG_PREFIX + ".config")
public class ShardingRuleConfigurationProperties extends YamlShardingRuleConfiguration {


}

同時sharding-jbc支持自定義一些properties屬性，需要單獨創建類ConfigMapConfigurationProperties

/**
 * @Author: Ship
 * @Description:
 * @Date: Created in 2023/6/6
 */
@ConfigurationProperties(prefix = CommonConstants.COMMON_CONFIG_PREFIX + ".map")
public class ConfigMapConfigurationProperties {

    private Properties props = new Properties();


    public Properties getProps() {
        return props;
    }

    public void setProps(Properties props) {
        this.props = props;
    }
}

官方提供了ShardingDataSourceFactory工廠類來創建數據源，但是查看其源碼發現createDataSource方法的參數是ShardingRuleConfiguration類，而不是YamlShardingRuleConfiguration。

@NoArgsConstructor(access = AccessLevel.PRIVATE)
public final class ShardingDataSourceFactory {
    
    /**
     * Create sharding data source.
     *
     * @param dataSourceMap data source map
     * @param shardingRuleConfig rule configuration for databases and tables sharding
     * @param props properties for data source
     * @return sharding data source
     * @throws SQLException SQL exception
     */
    public static DataSource createDataSource(
            final Map<String, DataSource> dataSourceMap, final ShardingRuleConfiguration shardingRuleConfig, final Properties props) throws SQLException {
        return new ShardingDataSource(dataSourceMap, new ShardingRule(shardingRuleConfig, dataSourceMap.keySet()), props);
    }
}

該如何解決配置類參數轉換的問題呢？

幸好查找官方文檔發現sharding-jdbc提供了YamlSwapper類來實現yaml配置和核心配置的轉換

/**
 * YAML configuration swapper.
 *
 * @param <Y> type of YAML configuration
 * @param <T> type of swapped object
 */
public interface YamlSwapper<Y extends YamlConfiguration, T> {
    
    /**
     * Swap to YAML configuration.
     *
     * @param data data to be swapped
     * @return YAML configuration
     */
    Y swap(T data);
    
    /**
     * Swap from YAML configuration to object.
     *
     * @param yamlConfiguration YAML configuration
     * @return swapped object
     */
    T swap(Y yamlConfiguration);
}

ShardingRuleConfigurationYamlSwapper就是YamlSwapper的其中一個實現類。

於是，ShardingAutoConfig的最終代碼如下：

package cn.sp.sharding.config;

/**
 * @author Ship
 * @version 1.0.0
 * @description:
 * @date 2023/06/06
 */
@AutoConfigureBefore(name = CommonConstants.MYBATIS_PLUS_CONFIG_CLASS)
@Configuration
@EnableConfigurationProperties(value = {ShardingRuleConfigurationProperties.class, ConfigMapConfigurationProperties.class})
@Import(DataSourceHealthConfig.class)
public class ShardingAutoConfig implements EnvironmentAware {


    private Map<String, DataSource> dataSourceMap = new HashMap<>();

    @ConditionalOnMissingBean
    @Bean
    public DataSource shardingDataSource(@Autowired ShardingRuleConfigurationProperties configurationProperties,
                                         @Autowired ConfigMapConfigurationProperties configMapConfigurationProperties) throws SQLException {
        ShardingRuleConfigurationYamlSwapper yamlSwapper = new ShardingRuleConfigurationYamlSwapper();
        ShardingRuleConfiguration shardingRuleConfiguration = yamlSwapper.swap(configurationProperties);
        return ShardingDataSourceFactory.createDataSource(dataSourceMap, shardingRuleConfiguration, configMapConfigurationProperties.getProps());
    }

    @Override
    public void setEnvironment(Environment environment) {
        setDataSourceMap(environment);
    }

    private void setDataSourceMap(Environment environment) {
        String names = environment.getProperty(CommonConstants.DATA_SOURCE_CONFIG_PREFIX + ".names");
        for (String name : names.split(",")) {
            try {
                String propertiesPrefix = CommonConstants.DATA_SOURCE_CONFIG_PREFIX + "." + name;
                Map<String, Object> dataSourceProps = PropertyUtil.handle(environment, propertiesPrefix, Map.class);
                // 反射創建數據源
                DataSource dataSource = DataSourceUtil.getDataSource(dataSourceProps.get("type").toString(), dataSourceProps);
                dataSourceMap.put(name, dataSource);
            } catch (ReflectiveOperationException e) {
                e.printStackTrace();
            } catch (Exception e) {
                e.printStackTrace();
            }
        }
    }
}

利用反射創建數據源，就可以解決支持多種數據源的問題。

3.3 自定義主鍵生成策略

sharding-jdbc提供了UUID和Snowflake兩種預設實現，但是自定義主鍵生成策略更加靈活，方便根據自己的需求調整，接下來介紹如何自定義主鍵生成策略。

因為我們也是用的雪花演算法，所以可以直接用sharding-jdbc提供的雪花演算法類，KeyGeneratorFactory負責生成雪花演算法實現類的實例，採用雙重校驗加鎖的單例模式。

public final class KeyGeneratorFactory {
    /**
     * 使用shardingsphere提供的雪花演算法實現
     */
    private static volatile SnowflakeShardingKeyGenerator keyGenerator = null;

    private KeyGeneratorFactory() {

    }

    /**
     * 單例模式
     *
     * @return
     */
    public static SnowflakeShardingKeyGenerator getInstance() {
        if (keyGenerator == null) {
            synchronized (KeyGeneratorFactory.class) {
                if (keyGenerator == null) {
                    // 用ip地址當作機器id，機器範圍0-1024
                    Long workerId = Long.valueOf(IpUtil.getLocalIpAddress().replace(".", "")) % 1024;
                    keyGenerator = new SnowflakeShardingKeyGenerator();
                    Properties properties = new Properties();
                    properties.setProperty("worker.id", workerId.toString());
                    keyGenerator.setProperties(properties);
                }
            }
        }
        return keyGenerator;
    }
}

雪花演算法是由1bit 不用 + 41bit時間戳+10bit工作機器id+12bit序列號組成的，所以為了防止不同節點生成的id重覆需要設置機器id，機器id的範圍是0-1024，這裡是用IP地址轉數字取模1024來電腦器id，存在很小概率的重覆，也可以用redis來生成機器id（參考雪花演算法ID重覆問題的解決方案 ）。

註意： 雪花演算法坑其實挺多的，除了系統時間回溯會導致id重覆，單節點併發過高也會導致重覆（序列位只有12位代表1ms內最多支持4096個併發）。

查看源碼可知自定義主鍵生成器是通過SPI實現的，實現ShardingKeyGenerator介面即可。

package org.apache.shardingsphere.spi.keygen;

import org.apache.shardingsphere.spi.TypeBasedSPI;

/**
 * Key generator.
 */
public interface ShardingKeyGenerator extends TypeBasedSPI {
    
    /**
     * Generate key.
     * 
     * @return generated key
     */
    Comparable<?> generateKey();
}

1. 自定義主鍵生成器DistributedKeyGenerator

/**
 * @Author: Ship
 * @Description: 分散式id生成器，雪花演算法實現
 * @Date: Created in 2023/6/8
 */
public class DistributedKeyGenerator implements ShardingKeyGenerator {

    @Override
    public Comparable<?> generateKey() {
        return KeyGeneratorFactory.getInstance().generateKey();
    }

    @Override
    public String getType() {
        return "DISTRIBUTED";
    }

    @Override
    public Properties getProperties() {
        return null;
    }

    @Override
    public void setProperties(Properties properties) {

    }
}

2. 創建META-INF/services文件夾，然後在文件夾下創建org.apache.shardingsphere.spi.keygen.ShardingKeyGenerator文件，內容如下：

 cn.sp.sharding.key.DistributedKeyGenerator

3. yaml文件配置即可

3.4 遺留問題

Spring Boot會在項目啟動時執行一條sql語句檢查數據源是否可用，因為ShardingDataSource只是對真實數據源進行了封裝，沒有完全實現Datasouce介面規範，所以會在啟動時報錯DataSource health check failed，為此需要重寫數據源健康檢查的邏輯。

創建DataSourceHealthConfig類繼承DataSourceHealthContributorAutoConfiguration，然後重寫createIndicator方法來重新設置校驗sql語句。

/**
 * @Author: Ship
 * @Description:
 * @Date: Created in 2023/6/7
 */
public class DataSourceHealthConfig extends DataSourceHealthContributorAutoConfiguration {

    private static String validQuery = "SELECT 1";

    public DataSourceHealthConfig(Map<String, DataSource> dataSources, ObjectProvider<DataSourcePoolMetadataProvider> metadataProviders) {
        super(dataSources, metadataProviders);
    }

    @Override
    protected AbstractHealthIndicator createIndicator(DataSource source) {
        DataSourceHealthIndicator healthIndicator = (DataSourceHealthIndicator) super.createIndicator(source);
        if (StringUtils.hasText(validQuery)) {
            healthIndicator.setQuery(validQuery);
        }
        return healthIndicator;
    }
}

最後使用@Import註解來註入

@AutoConfigureBefore(name = CommonConstants.MYBATIS_PLUS_CONFIG_CLASS)
@Configuration
@EnableConfigurationProperties(value = {ShardingRuleConfigurationProperties.class, ConfigMapConfigurationProperties.class})
@Import(DataSourceHealthConfig.class)
public class ShardingAutoConfig implements EnvironmentAware {

四、測試

假設有個訂單表數據量很大了需要分表，為了方便水平擴展，根據訂單的創建時間分表，分表規則如下：

t_order_${創建時間所在年}_${創建時間所在季度}

訂單表結構如下

CREATE TABLE `t_order_2022_3` (
  `id` bigint(20) unsigned NOT NULL COMMENT '主鍵',
  `order_code` varchar(32) DEFAULT NULL COMMENT '訂單號',
  `create_time` bigint(20) NOT NULL COMMENT '創建時間',
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

1. 創建資料庫my_springboot,並創建8張訂單表t_order_2022_1至t_order_2023_4

2. 創建SpringBoot項目ship-sharding-example，並添加mybatis等相關依賴

  <dependency>
            <groupId>org.mybatis.spring.boot</groupId>
            <artifactId>mybatis-spring-boot-starter</artifactId>
            <version>${mybatis.version}</version>
        </dependency>


        <dependency>
            <groupId>com.baomidou</groupId>
            <artifactId>mybatis-plus-boot-starter</artifactId>
            <version>3.0.1</version>
            <exclusions>
                <exclusion>
                    <groupId>org.mybatis</groupId>
                    <artifactId>mybatis</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        
        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>druid-spring-boot-starter</artifactId>
            <version>${druid.version}</version>
        </dependency>

        <dependency>
            <groupId>cn.sp</groupId>
            <artifactId>ship-sharding-spring-boot-starter</artifactId>
            <version>1.0-SNAPSHOT</version>
        </dependency>

        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
        </dependency>

3. 創建訂單實體Order和OrderMapper,代碼比較簡單省略
4. 自定義分表演算法需要實現PreciseShardingAlgorithm和RangeShardingAlgorithm介面的方法，它倆區別如下

創建演算法類MyTableShardingAlgorithm

/**
 * @Author: Ship
 * @Description:
 * @Date: Created in 2023/6/8
 */
@Slf4j
public class MyTableShardingAlgorithm implements PreciseShardingAlgorithm<Long>, RangeShardingAlgorithm<Long> {

    private static final String TABLE_NAME_PREFIX = "t_order_";

    @Override
    public String doSharding(Collection<String> availableTableNames, PreciseShardingValue<Long> preciseShardingValue) {
        Long createTime = preciseShardingValue.getValue();
        if (createTime == null) {
            throw new ShipShardingException("創建時間不能為空！");
        }
        LocalDate localDate = DateUtils.longToLocalDate(createTime);
        final String year = localDate.getYear() + "";
        Integer quarter = DateUtils.getQuarter(localDate);
        for (String tableName : availableTableNames) {
            String dateStr = tableName.replace(TABLE_NAME_PREFIX, "");
            String[] dateArr = dateStr.split("_");
            if (dateArr[0].equals(year) && dateArr[1].equals(quarter.toString())) {
                return tableName;
            }
        }
        log.error("分表演算法對應的表不存在！");
        throw new ShipShardingException("分表演算法對應的表不存在！");
    }

    @Override
    public Collection<String> doSharding(Collection<String> availableTableNames, RangeShardingValue<Long> rangeShardingValue) {
        //獲取查詢條件中範圍值
        Range<Long> valueRange = rangeShardingValue.getValueRange();
        // 上限值
        Long upperEndpoint = valueRange.upperEndpoint();
        // 下限值
        Long lowerEndpoint = valueRange.lowerEndpoint();

        List<String> tableNames = Lists.newArrayList();
        for (String tableName : availableTableNames) {
            String dateStr = tableName.replace(MyTableShardingAlgorithm.TABLE_NAME_PREFIX, "");
            String[] dateArr = dateStr.split("_");
            String year = dateArr[0];
            String quarter = dateArr[1];
            Long[] minAndMaxTime = DateUtils.getMinAndMaxTime(year, quarter);
            Long minTime = minAndMaxTime[0];
            Long maxTime = minAndMaxTime[1];
            if (valueRange.hasLowerBound() && valueRange.hasUpperBound()) {
                // between and
                if (minTime.compareTo(lowerEndpoint) <= 0 && upperEndpoint.compareTo(maxTime) <= 0) {
                    tableNames.add(tableName);
                }
            } else if (valueRange.hasLowerBound() && !valueRange.hasUpperBound()) {
                if (maxTime.compareTo(lowerEndpoint) > 0) {
                    tableNames.add(tableName);
                }
            } else {
                if (upperEndpoint.compareTo(minTime) > 0) {
                    tableNames.add(tableName);
                }
            }
        }
        if (tableNames.size() == 0) {
            log.error("分表演算法對應的表不存在！");
            throw new ShipShardingException("分表演算法對應的表不存在！");
        }
        return tableNames;
    }
}

5. 在application.yaml上添加資料庫配置和分表配置

spring:
  application:
    name: ship-sharding-example


mybatis-plus:
  base-package: cn.sp.sharding.dao
  mapper-locations: classpath*:/mapper/*Mapper.xml
  configuration:
    #開啟自動駝峰命名規則（camel case）映射
    map-underscore-to-camel-case: true
    #延遲載入，需要和lazy-loading-enabled一起使用
    aggressive-lazy-loading: true
    lazy-loading-enabled: true
    #關閉一級緩存
    local-cache-scope: statement
    #關閉二級級緩存
    cache-enabled: false

ship:
  sharding:
    jdbc:
      datasource:
        names: ds0
        ds0:
          driver-class-name: com.mysql.cj.jdbc.Driver
          type: com.alibaba.druid.pool.DruidDataSource
          url: jdbc:mysql://127.0.0.1:3306/my_springboot?autoReconnect=true&useUnicode=true&characterEncoding=UTF-8&allowMultiQueries=true&useSSL=false
          username: root
          password: 1234
          initial-size: 5
          minIdle: 5
          maxActive: 20
          maxWait: 60000
          timeBetweenEvictionRunsMillis: 60000
          minEvictableIdleTimeMillis: 300000
          validationQuery: SELECT 1 FROM DUAL
          testWhileIdle: true
          testOnBorrow: false
          testOnReturn: false
          poolPreparedStatements: true
          maxPoolPreparedStatementPerConnectionSize: 20
          useGlobalDataSourceStat: true
          connectionProperties: druid.stat.mergeSql=true;druid.stat.slowSqlMillis=2000;druid.mysql.usePingMethod=false
      config:
        binding-tables: t_order
        tables:
          t_order:
            actual-data-nodes: ds0.t_order_${2022..2023}_${1..4}
            # 配置主鍵生成策略
            key-generator:
              type: DISTRIBUTED
              column: id
            table-strategy:
              standard:
                sharding-column: create_time
                # 配置分表演算法
                precise-algorithm-class-name: cn.sp.sharding.algorithm.MyTableShardingAlgorithm
                range-algorithm-class-name: cn.sp.sharding.algorithm.MyTableShardingAlgorithm

6. 現在可以進行測試了，首先寫一個單元測試測試數據插入情況。

 @Test
    public void testInsert() {
        Order order = new Order();
        order.setOrderCode("OC001");
        order.setCreateTime(System.currentTimeMillis());
        orderMapper.insert(order);
    }

運行testInsert()方法，打開t_order_2023_2表發現已經有了一條訂單數據

並且該數據的create_time是1686383781371，轉換為時間為2023-06-10 15:56:21，剛好對應2023年第二季度，說明數據正確的路由到了對應的表裡。

然後測試下數據查詢情況

@Test
    public void testQuery(){
        QueryWrapper<Order> wrapper = new QueryWrapper<>();
        wrapper.lambda().eq(Order::getOrderCode,"OC001");
        List<Order> orders = orderMapper.selectList(wrapper);
        System.out.println(JSONUtil.toJsonStr(orders));
    }

運行testQuery()方法後可以在控制台看到輸出了訂單報文，說明查詢也沒問題。

[{"id":1667440550397132802,"orderCode":"OC001","createTime":1686383781371}]

五、總結

本文代碼已經上傳到github，後續會把ship-sharding-spring-boot-starter上傳到maven中央倉庫方便使用，如果覺得對你有用的話希望可以點個贊讓更多人看到