背景 Redis多數據源常見的場景: 分區數據處理:當數據量增長時,單個Redis實例可能無法處理所有的數據。通過使用多個Redis數據源,可以將數據分區存儲在不同的實例中,使得數據處理更加高效。 多租戶應用程式:對於多租戶應用程式,每個租戶可以擁有自己的Redis數據源,以確保數據隔離和安全性。 ...
背景
Redis多數據源常見的場景:
- 分區數據處理:當數據量增長時,單個Redis實例可能無法處理所有的數據。通過使用多個Redis數據源,可以將數據分區存儲在不同的實例中,使得數據處理更加高效。
- 多租戶應用程式:對於多租戶應用程式,每個租戶可以擁有自己的Redis數據源,以確保數據隔離和安全性。
目前在網上的一些解決方案無法完全滿足特定需求,例如不支持動態添加或更新數據源、缺乏數據源的負載均衡功能,或者不能開箱即用需要自行封裝一些常用方法。
為瞭解決這些問題,封裝了一個輕量級的Redis多數據源庫,已在生產環境中長時間穩定運行。現分享出來,希望這個庫能對你的項目有所幫助。
https://github.com/codebaorg/redis-keeper
如果這篇文章幫助到了你,歡迎star一下,感謝你的支持。
特征
- 基於redisson封裝,保留redisson的所有強大功能。
- 支持redis多數據源配置的實時更新。
- 支持redis多數據源的負載均衡。
- 支持redis數據源的“只讀”、“只寫”、“讀寫”、“跳過”的狀態切換。
- 提供 130+ 常用的redis操作方法,包括但不限string、list、hash、set、zset、geo、bitmap、hyberloglog、分散式鎖、布隆過濾器、限流等。
快速開始
- 添加maven依賴
<dependency>
<groupId>org.redisson</groupId>
<artifactId>redisson</artifactId>
<version>3.15.5</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.codeba</groupId>
<artifactId>redis-keeper-spring-boot-starter</artifactId>
<version>2024.0.0</version>
</dependency>
- springboot的配置文件,添加redis多數據源的配置
redis-keeper:
redis:
datasource:
ds1: // 此處的命名可以隨便取,但是要保證唯一不重覆
host: localhost
port: 6379
database: 0
password: yourPass
ds2: // 此處的命名可以隨便取,但是要保證唯一不重覆
host: localhost
port: 6379
database: 0
password: yourPass
- 按自定義的數據源別名獲取相應的cacheTemplate,執行redis命令。
方式一:通過CacheTemplateProvider獲取cacheTemplate,示例如下:
public class AppTest {
@Autowired
private CacheTemplateProvider<CacheTemplate> provider;
public void test() {
final Optional<CacheTemplate> templateOptional = provider.getTemplate("ds1");
if (templateOptional.isPresent()) {
final CacheTemplate cacheTemplate = templateOptional.get();
String key = "foo";
String value = "bar";
cacheTemplate.set(key, value);
}
}
}
方式二:通過代理類CacheTemplateProxy獲取cacheTemplate,示例如下:
public class AppTest {
private final CacheTemplate cacheTemplate = CacheTemplateProxy.asTemplate("ds1");
public void test() {
String key = "foo";
String value = "bar";
cacheTemplate.set(key, value);
}
}
更多示例
多數據源實時更新
需要:spring cloud + nacos
現有的數據源配置示例如下:
redis-keeper:
redis:
datasource:
ds1:
host: localhost
port: 6379
database: 10
password: xxx
invoke-params-print: true
測試方法:
@RestController
@RequiredArgsConstructor
public class TestController {
private final CacheTemplateProvider<CacheTemplate> provider;
@RequestMapping("/test")
public void testRefresh() {
final Optional<CacheTemplate> templateOptional = provider.getTemplate("ds1");
templateOptional.ifPresent(cacheTemplate -> {
cacheTemplate.set("foo", "bar");
cacheTemplate.del("foo");
});
}
}
訪問上面的方法,日誌列印如下(可以看到寫入的redis庫是10庫):
2024-03-12T11:33:37.850+08:00 INFO 62914 --- [nio-8000-exec-1] o.c.r.k.support.DefaultRedissonTemplate : cmd:set, params:[foo, bar], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=10}]
2024-03-12T11:33:37.858+08:00 INFO 62914 --- [nio-8000-exec-1] o.c.r.k.support.DefaultRedissonTemplate : cmd:del, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=10}]
修改數據源配置,比如修改庫為0庫,此時可以看到nacos配置實時更新日誌:
2024-03-12T11:43:51.683+08:00 INFO 62914 --- [r-localhost-216] com.alibaba.nacos.common.remote.client : [6253dca3-136e-4ac1-b894-fb4c11483703_config-0] Receive server push request, request = ConfigChangeNotifyRequest, requestId = 1
2024-03-12T11:43:51.683+08:00 INFO 62914 --- [r-localhost-216] c.a.n.client.config.impl.ClientWorker : [6253dca3-136e-4ac1-b894-fb4c11483703_config-0] [server-push] config changed. dataId=redis-keeper-demo.yaml, group=DEFAULT_GROUP,tenant=null
2024-03-12T11:43:51.684+08:00 INFO 62914 --- [r-localhost-216] com.alibaba.nacos.common.remote.client : [6253dca3-136e-4ac1-b894-fb4c11483703_config-0] Ack server push request, request = ConfigChangeNotifyRequest, requestId = 1
2024-03-12T11:43:51.700+08:00 INFO 62914 --- [s.client.Worker] c.a.n.client.config.impl.ClientWorker : [fixed-localhost_8848] [data-received] dataId=redis-keeper-demo.yaml, group=DEFAULT_GROUP, tenant=, md5=dddd513021564c6d6322e18783ac4f2b, content=server:
redis-keeper:
redis:
datasource:
ds1:
host: localhost
..., type=yaml
2024-03-12T11:43:51.701+08:00 INFO 62914 --- [s.client.Worker] c.a.nacos.client.config.impl.CacheData : [fixed-localhost_8848] [notify-listener] time cost=0ms in ClientWorker, dataId=redis-keeper-demo.yaml, group=DEFAULT_GROUP, md5=dddd513021564c6d6322e18783ac4f2b, listener=com.alibaba.cloud.nacos.refresh.NacosContextRefresher$1@178412d1
2024-03-12T11:43:51.701+08:00 INFO 62914 --- [ternal.notifier] c.a.nacos.client.config.impl.CacheData : [fixed-localhost_8848] [notify-context] dataId=redis-keeper-demo.yaml, group=DEFAULT_GROUP, md5=dddd513021564c6d6322e18783ac4f2b
2024-03-12T11:43:51.915+08:00 INFO 62914 --- [ternal.notifier] c.a.c.n.c.NacosConfigDataLoader : [Nacos Config] Load config[dataId=redis-keeper-demo.yaml, group=DEFAULT_GROUP] success
2024-03-12T11:43:51.950+08:00 INFO 62914 --- [ternal.notifier] o.s.c.e.event.RefreshEventListener : Refresh keys changed: [redis-keeper.redis.datasource.ds1.database]
2024-03-12T11:43:51.950+08:00 INFO 62914 --- [ternal.notifier] c.a.nacos.client.config.impl.CacheData : [fixed-localhost_8848] [notify-ok] dataId=redis-keeper-demo.yaml, group=DEFAULT_GROUP, md5=dddd513021564c6d6322e18783ac4f2b, listener=com.alibaba.cloud.nacos.refresh.NacosContextRefresher$1@178412d1 ,cost=249 millis.
再次訪問上面的方法,日誌列印如下(可以看到寫入的redis庫是0庫):
2024-03-12T11:46:11.849+08:00 INFO 62914 --- [nio-8000-exec-4] o.c.r.k.support.DefaultRedissonTemplate : cmd:set, params:[foo, bar], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=0}]
2024-03-12T11:46:11.861+08:00 INFO 62914 --- [nio-8000-exec-4] o.c.r.k.support.DefaultRedissonTemplate : cmd:del, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=0}]
多數據源負載均衡
有的業務場景需要多套redis集群,來承接同一業務的流量,起到分流的作用,這時就需要從多個數據源中獲取其中的某一個進行redis的讀寫。
負載均衡的演算法目前支持兩種:輪詢、隨機。
redis多數據源配置,示例如下:
註意:只有 redis-keeper.redis.datasources 或者 redis-keeper.redisson.datasources 配置下的多數據源支持負載均衡。
redis-keeper:
redis:
datasources:
ds2:
- host: localhost
port: 6379
database: 1
password: xxx
invoke-params-print: true
- host: localhost
port: 6379
database: 2
password: xxx
invoke-params-print: true
- host: localhost
port: 6379
database: 3
password: xxx
invoke-params-print: true
輪詢示例如下:
public class AppTest {
@Autowired
private CacheTemplateProvider<CacheTemplate> provider;
public void testPollTemplate() {
String key = "foo";
for (int i = 1; i <= 10; i++) {
final Optional<CacheTemplate> templateOptional = provider.pollTemplate("ds2");
if (templateOptional.isPresent()) {
final CacheTemplate template = templateOptional.get();
final String str = String.valueOf(i);
template.set(key, str);
template.get(key).ifPresent(el -> {
assert str.equals(el);
});
template.del(key);
}
}
}
}
日誌列印結果,可以看到配置的redis數據源的出現的次數是相近的:
2024-03-12 12:15:15.614 INFO 67575 --- [main] : cmd:set, params:[foo, 1], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=1}]
2024-03-12 12:15:15.621 INFO 67575 --- [main] : cmd:get, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=1}]
2024-03-12 12:15:15.624 INFO 67575 --- [main] : cmd:del, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=1}]
2024-03-12 12:15:15.627 INFO 67575 --- [main] : cmd:set, params:[foo, 2], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=2}]
2024-03-12 12:15:15.629 INFO 67575 --- [main] : cmd:get, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=2}]
2024-03-12 12:15:15.635 INFO 67575 --- [main] : cmd:del, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=2}]
2024-03-12 12:15:15.635 INFO 67575 --- [main] : cmd:set, params:[foo, 3], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]
2024-03-12 12:15:15.636 INFO 67575 --- [main] : cmd:get, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]
2024-03-12 12:15:15.638 INFO 67575 --- [main] : cmd:del, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]
2024-03-12 12:15:15.639 INFO 67575 --- [main] : cmd:set, params:[foo, 4], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=1}]
2024-03-12 12:15:15.652 INFO 67575 --- [main] : cmd:get, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=1}]
2024-03-12 12:15:15.653 INFO 67575 --- [main] : cmd:del, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=1}]
2024-03-12 12:15:15.654 INFO 67575 --- [main] : cmd:set, params:[foo, 5], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=2}]
2024-03-12 12:15:15.655 INFO 67575 --- [main] : cmd:get, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=2}]
2024-03-12 12:15:15.656 INFO 67575 --- [main] : cmd:del, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=2}]
2024-03-12 12:15:15.656 INFO 67575 --- [main] : cmd:set, params:[foo, 6], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]
2024-03-12 12:15:15.658 INFO 67575 --- [main] : cmd:get, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]
2024-03-12 12:15:15.658 INFO 67575 --- [main] : cmd:del, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]
2024-03-12 12:15:15.659 INFO 67575 --- [main] : cmd:set, params:[foo, 7], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=1}]
2024-03-12 12:15:15.660 INFO 67575 --- [main] : cmd:get, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=1}]
2024-03-12 12:15:15.661 INFO 67575 --- [main] : cmd:del, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=1}]
2024-03-12 12:15:15.667 INFO 67575 --- [main] : cmd:set, params:[foo, 8], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=2}]
2024-03-12 12:15:15.669 INFO 67575 --- [main] : cmd:get, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=2}]
2024-03-12 12:15:15.669 INFO 67575 --- [main] : cmd:del, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=2}]
2024-03-12 12:15:15.670 INFO 67575 --- [main] : cmd:set, params:[foo, 9], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]
2024-03-12 12:15:15.672 INFO 67575 --- [main] : cmd:get, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]
2024-03-12 12:15:15.673 INFO 67575 --- [main] : cmd:del, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]
2024-03-12 12:15:15.674 INFO 67575 --- [main] : cmd:set, params:[foo, 10], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=1}]
2024-03-12 12:15:15.675 INFO 67575 --- [main] : cmd:get, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=1}]
2024-03-12 12:15:15.676 INFO 67575 --- [main] : cmd:del, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=1}]
隨機示例如下:
public class AppTest {
@Autowired
private CacheTemplateProvider<CacheTemplate> provider;
public void testRandomTemplate() {
String key = "foo";
for (int i = 1; i <= 10; i++) {
final Optional<CacheTemplate> templateOptional = provider.randomTemplate("ds2");
if (templateOptional.isPresent()) {
final CacheTemplate template = templateOptional.get();
final String str = String.valueOf(i);
template.set(key, str);
template.get(key).ifPresent(el -> {
assert str.equals(el);
});
template.del(key);
}
}
}
}
日誌列印結果,可以看到配置的redis數據源的出現的次數是隨機的:
2024-03-12 12:05:06.450 INFO 66199 --- [main]: cmd:set, params:[foo, 1], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]
2024-03-12 12:05:06.457 INFO 66199 --- [main]: cmd:get, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]
2024-03-12 12:05:06.461 INFO 66199 --- [main]: cmd:del, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]
2024-03-12 12:05:06.464 INFO 66199 --- [main]: cmd:set, params:[foo, 2], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]
2024-03-12 12:05:06.465 INFO 66199 --- [main]: cmd:get, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]
2024-03-12 12:05:06.467 INFO 66199 --- [main]: cmd:del, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]
2024-03-12 12:05:06.468 INFO 66199 --- [main]: cmd:set, params:[foo, 3], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]
2024-03-12 12:05:06.469 INFO 66199 --- [main]: cmd:get, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]
2024-03-12 12:05:06.470 INFO 66199 --- [main]: cmd:del, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]
2024-03-12 12:05:06.472 INFO 66199 --- [main]: cmd:set, params:[foo, 4], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=2}]
2024-03-12 12:05:06.473 INFO 66199 --- [main]: cmd:get, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=2}]
2024-03-12 12:05:06.475 INFO 66199 --- [main]: cmd:del, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=2}]
2024-03-12 12:05:06.475 INFO 66199 --- [main]: cmd:set, params:[foo, 5], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=1}]
2024-03-12 12:05:06.476 INFO 66199 --- [main]: cmd:get, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=1}]
2024-03-12 12:05:06.477 INFO 66199 --- [main]: cmd:del, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=1}]
2024-03-12 12:05:06.478 INFO 66199 --- [main]: cmd:set, params:[foo, 6], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]
2024-03-12 12:05:06.479 INFO 66199 --- [main]: cmd:get, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]
2024-03-12 12:05:06.481 INFO 66199 --- [main]: cmd:del, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]
2024-03-12 12:05:06.482 INFO 66199 --- [main]: cmd:set, params:[foo, 7], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]
2024-03-12 12:05:06.484 INFO 66199 --- [main]: cmd:get, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]
2024-03-12 12:05:06.485 INFO 66199 --- [main]: cmd:del, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]
2024-03-12 12:05:06.486 INFO 66199 --- [main]: cmd:set, params:[foo, 8], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]
2024-03-12 12:05:06.488 INFO 66199 --- [main]: cmd:get, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]
2024-03-12 12:05:06.489 INFO 66199 --- [main]: cmd:del, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]
2024-03-12 12:05:06.491 INFO 66199 --- [main]: cmd:set, params:[foo, 9], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=2}]
2024-03-12 12:05:06.492 INFO 66199 --- [main]: cmd:get, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=2}]
2024-03-12 12:05:06.493 INFO 66199 --- [main]: cmd:del, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=2}]
2024-03-12 12:05:06.494 INFO 66199 --- [main]: cmd:set, params:[foo, 10], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]
2024-03-12 12:05:06.495 INFO 66199 --- [main]: cmd:get, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]
2024-03-12 12:05:06.496 INFO 66199 --- [main]: cmd:del, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]
獲取指定狀態標識的數據源
預設情況下數據源的狀態標識是讀寫,還有三個可配狀態標識:只讀、只寫、跳過。
註意:此處的狀態只是一種標識,用於篩選獲取指定的數據源,並不用於控制redis的讀寫命令。
數據源配置如下:
redis-keeper:
redis:
datasource:
ds1:
host: localhost
port: 6379
database: 10
password: xxx
status: RW
invoke-params-print: true
測試方法:
public void testTemplateWithStatus() {
final Optional<CacheTemplate> templateOptional = provider.getTemplate("ds1", CacheDatasourceStatus.RW);
if (templateOptional.isPresent()) {
final CacheTemplate cacheTemplate = templateOptional.get();
String key = "foo";
String value = "bar";
cacheTemplate.set(key, value);
final Optional<Object> optional = cacheTemplate.get(key);
optional.ifPresent(el -> {
assert value.equals(el);
});
cacheTemplate.del(key);
}
assert !provider.getTemplate("ds1", CacheDatasourceStatus.SKIP).isPresent();
}
訪問上面的方法,日誌列印如下:
2024-03-12 14:28:34.183 INFO 77230 --- [main]: cmd:set, params:[foo, bar], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=0}]
2024-03-12 14:28:34.190 INFO 77230 --- [main]: cmd:get, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=0}]
2024-03-12 14:28:34.193 INFO 77230 --- [main]: cmd:del, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=0}]
更多詳細示例,請查看:https://github.com/codebaorg/redis-keeper/blob/main/README_zh.md
對比其他方案
- https://cloud.tencent.com/developer/article/2195679
- https://juejin.cn/post/7012164339255738399
- https://www.cnblogs.com/Zzzyyw/p/17116721.html
- https://learnku.com/articles/37188
- https://iogogogo.github.io/2020/01/10/spring-boot-redis-multi-instance/
如果這篇文章幫助到了你,歡迎評論、點贊、轉發。
本文由博客一文多發平臺 OpenWrite 發佈!