" 【.NET Core項目實戰 統一認證平臺】開篇及目錄索引 " 這篇文章,我們將從Ocelot的中間件源碼分析,目前Ocelot已經實現那些功能,還有那些功能在我們實際項目中暫時還未實現,如果我們要使用這些功能,應該如何改造等方面來說明。 一、Ocelot源碼解讀 在使用一個組件前,最好我們要了 ...
【.NET Core項目實戰-統一認證平臺】開篇及目錄索引
這篇文章,我們將從Ocelot的中間件源碼分析,目前Ocelot已經實現那些功能,還有那些功能在我們實際項目中暫時還未實現,如果我們要使用這些功能,應該如何改造等方面來說明。
一、Ocelot源碼解讀
在使用一個組件前,最好我們要瞭解其中的一些原理,否則在使用過程中遇到問題,也無從下手,今天我帶著大家一起來解讀下Ocelot源碼,並梳理出具體實現的原理和流程,便於我們根據需求擴展應用。
Ocelot源碼地址[https://github.com/ThreeMammals/Ocelot],
Ocelot文檔地址[https://ocelot.readthedocs.io/en/latest/]
查看.NETCORE
相關中間件源碼,我們優先找到入口方法,比如Ocelot中間件使用的是app.UseOcelot()
,我們直接搜索UserOcelot,我們會找到OcelotMiddlewareExtensions
方法,裡面是Ocelot中間件實際運行的方式和流程。
然後繼續順藤摸瓜,查看詳細的實現,我們會發現如下代碼
public static async Task<IApplicationBuilder> UseOcelot(this IApplicationBuilder builder, OcelotPipelineConfiguration pipelineConfiguration)
{
//創建配置信息
var configuration = await CreateConfiguration(builder);
//監聽配置信息
ConfigureDiagnosticListener(builder);
//創建執行管道
return CreateOcelotPipeline(builder, pipelineConfiguration);
}
然後我們繼續跟蹤到創建管道方法,可以發現Ocelot的執行流程已經被找到,現在問題變的簡單了,直接查看
private static IApplicationBuilder CreateOcelotPipeline(IApplicationBuilder builder, OcelotPipelineConfiguration pipelineConfiguration)
{
var pipelineBuilder = new OcelotPipelineBuilder(builder.ApplicationServices);
//詳細創建的管道順序在此方法
pipelineBuilder.BuildOcelotPipeline(pipelineConfiguration);
var firstDelegate = pipelineBuilder.Build();
/*
inject first delegate into first piece of asp.net middleware..maybe not like this
then because we are updating the http context in ocelot it comes out correct for
rest of asp.net..
*/
builder.Properties["analysis.NextMiddlewareName"] = "TransitionToOcelotMiddleware";
builder.Use(async (context, task) =>
{
var downstreamContext = new DownstreamContext(context);
await firstDelegate.Invoke(downstreamContext);
});
return builder;
}
管道創建流程及實現,會不會感覺到摸到大動脈了,核心的功能及原理基本找到了,那以後動手術也就可以避開一些坑了,我們可以對著這個執行順序,再查看詳細的源碼,按照這個執行順序查看源碼,您就會發現整個思路非常清晰,每一步的實現一目瞭然。為了更直觀的介紹源碼的解讀方式,這裡我們就拿我們後續要操刀的中間件來講解下中間件的具體實現。
public static class OcelotPipelineExtensions
{
public static OcelotRequestDelegate BuildOcelotPipeline(this IOcelotPipelineBuilder builder,
OcelotPipelineConfiguration pipelineConfiguration)
{
// This is registered to catch any global exceptions that are not handled
// It also sets the Request Id if anything is set globally
builder.UseExceptionHandlerMiddleware();
// If the request is for websockets upgrade we fork into a different pipeline
builder.MapWhen(context => context.HttpContext.WebSockets.IsWebSocketRequest,
app =>
{
app.UseDownstreamRouteFinderMiddleware();
app.UseDownstreamRequestInitialiser();
app.UseLoadBalancingMiddleware();
app.UseDownstreamUrlCreatorMiddleware();
app.UseWebSocketsProxyMiddleware();
});
// Allow the user to respond with absolutely anything they want.
builder.UseIfNotNull(pipelineConfiguration.PreErrorResponderMiddleware);
// This is registered first so it can catch any errors and issue an appropriate response
builder.UseResponderMiddleware();
// Then we get the downstream route information
builder.UseDownstreamRouteFinderMiddleware();
// This security module, IP whitelist blacklist, extended security mechanism
builder.UseSecurityMiddleware();
//Expand other branch pipes
if (pipelineConfiguration.MapWhenOcelotPipeline != null)
{
foreach (var pipeline in pipelineConfiguration.MapWhenOcelotPipeline)
{
builder.MapWhen(pipeline);
}
}
// Now we have the ds route we can transform headers and stuff?
builder.UseHttpHeadersTransformationMiddleware();
// Initialises downstream request
builder.UseDownstreamRequestInitialiser();
// We check whether the request is ratelimit, and if there is no continue processing
builder.UseRateLimiting();
// This adds or updates the request id (initally we try and set this based on global config in the error handling middleware)
// If anything was set at global level and we have a different setting at re route level the global stuff will be overwritten
// This means you can get a scenario where you have a different request id from the first piece of middleware to the request id middleware.
builder.UseRequestIdMiddleware();
// Allow pre authentication logic. The idea being people might want to run something custom before what is built in.
builder.UseIfNotNull(pipelineConfiguration.PreAuthenticationMiddleware);
// Now we know where the client is going to go we can authenticate them.
// We allow the ocelot middleware to be overriden by whatever the
// user wants
if (pipelineConfiguration.AuthenticationMiddleware == null)
{
builder.UseAuthenticationMiddleware();
}
else
{
builder.Use(pipelineConfiguration.AuthenticationMiddleware);
}
// The next thing we do is look at any claims transforms in case this is important for authorisation
builder.UseClaimsToClaimsMiddleware();
// Allow pre authorisation logic. The idea being people might want to run something custom before what is built in.
builder.UseIfNotNull(pipelineConfiguration.PreAuthorisationMiddleware);
// Now we have authenticated and done any claims transformation we
// can authorise the request
// We allow the ocelot middleware to be overriden by whatever the
// user wants
if (pipelineConfiguration.AuthorisationMiddleware == null)
{//使用自定義認證,移除預設的認證方式
//builder.UseAuthorisationMiddleware();
}
else
{
builder.Use(pipelineConfiguration.AuthorisationMiddleware);
}
// Now we can run the claims to headers transformation middleware
builder.UseClaimsToHeadersMiddleware();
// Allow the user to implement their own query string manipulation logic
builder.UseIfNotNull(pipelineConfiguration.PreQueryStringBuilderMiddleware);
// Now we can run any claims to query string transformation middleware
builder.UseClaimsToQueryStringMiddleware();
// Get the load balancer for this request
builder.UseLoadBalancingMiddleware();
// This takes the downstream route we retrieved earlier and replaces any placeholders with the variables that should be used
builder.UseDownstreamUrlCreatorMiddleware();
// Not sure if this is the best place for this but we use the downstream url
// as the basis for our cache key.
builder.UseOutputCacheMiddleware();
//We fire off the request and set the response on the scoped data repo
builder.UseHttpRequesterMiddleware();
return builder.Build();
}
private static void UseIfNotNull(this IOcelotPipelineBuilder builder,
Func<DownstreamContext, Func<Task>, Task> middleware)
{
if (middleware != null)
{
builder.Use(middleware);
}
}
}
限流中間件實現解析
實現代碼如下builder.UseRateLimiting();
,我們轉到定義,得到如下代碼,詳細的實現邏輯在ClientRateLimitMiddleware
方法里,繼續轉定義到這個方法,我把方法里用到的內容註釋了下。
public static class RateLimitMiddlewareExtensions
{
public static IOcelotPipelineBuilder UseRateLimiting(this IOcelotPipelineBuilder builder)
{
return builder.UseMiddleware<ClientRateLimitMiddleware>();
}
}
public class ClientRateLimitMiddleware : OcelotMiddleware
{
private readonly OcelotRequestDelegate _next;
private readonly IRateLimitCounterHandler _counterHandler;
private readonly ClientRateLimitProcessor _processor;
public ClientRateLimitMiddleware(OcelotRequestDelegate next,
IOcelotLoggerFactory loggerFactory,
IRateLimitCounterHandler counterHandler)
:base(loggerFactory.CreateLogger<ClientRateLimitMiddleware>())
{
_next = next;
_counterHandler = counterHandler;
_processor = new ClientRateLimitProcessor(counterHandler);
}
//熟悉的Tnvoke方法,所有的邏輯都在此方法里。
public async Task Invoke(DownstreamContext context)
{
var options = context.DownstreamReRoute.RateLimitOptions;
// 校驗是否啟用限流配置
if (!context.DownstreamReRoute.EnableEndpointEndpointRateLimiting)
{//未啟用直接進入下一個中間件
Logger.LogInformation($"EndpointRateLimiting is not enabled for {context.DownstreamReRoute.DownstreamPathTemplate.Value}");
await _next.Invoke(context);
return;
}
// 獲取配置的校驗客戶端的方式
var identity = SetIdentity(context.HttpContext, options);
// 校驗是否為白名單
if (IsWhitelisted(identity, options))
{//白名單直接放行
Logger.LogInformation($"{context.DownstreamReRoute.DownstreamPathTemplate.Value} is white listed from rate limiting");
await _next.Invoke(context);
return;
}
var rule = options.RateLimitRule;
if (rule.Limit > 0)
{//限流數是否大於0
// 獲取當前客戶端請求情況,這裡需要註意_processor是從哪裡註入的,後續重
var counter = _processor.ProcessRequest(identity, options);
// 校驗請求數是否大於限流數
if (counter.TotalRequests > rule.Limit)
{
//獲取下次有效請求的時間,就是避免每次請求,都校驗一次
var retryAfter = _processor.RetryAfterFrom(counter.Timestamp, rule);
// 寫入日誌
LogBlockedRequest(context.HttpContext, identity, counter, rule, context.DownstreamReRoute);
var retrystring = retryAfter.ToString(System.Globalization.CultureInfo.InvariantCulture);
// 拋出超出限流異常並把下次可請求時間寫入header里。
await ReturnQuotaExceededResponse(context.HttpContext, options, retrystring);
return;
}
}
//如果啟用了限流頭部
if (!options.DisableRateLimitHeaders)
{
var headers = _processor.GetRateLimitHeaders(context.HttpContext, identity, options);
context.HttpContext.Response.OnStarting(SetRateLimitHeaders, state: headers);
}
//進入下一個中間件
await _next.Invoke(context);
}
public virtual ClientRequestIdentity SetIdentity(HttpContext httpContext, RateLimitOptions option)
{
var clientId = "client";
if (httpContext.Request.Headers.Keys.Contains(option.ClientIdHeader))
{
clientId = httpContext.Request.Headers[option.ClientIdHeader].First();
}
return new ClientRequestIdentity(
clientId,
httpContext.Request.Path.ToString().ToLowerInvariant(),
httpContext.Request.Method.ToLowerInvariant()
);
}
public bool IsWhitelisted(ClientRequestIdentity requestIdentity, RateLimitOptions option)
{
if (option.ClientWhitelist.Contains(requestIdentity.ClientId))
{
return true;
}
return false;
}
public virtual void LogBlockedRequest(HttpContext httpContext, ClientRequestIdentity identity, RateLimitCounter counter, RateLimitRule rule, DownstreamReRoute downstreamReRoute)
{
Logger.LogInformation(
$"Request {identity.HttpVerb}:{identity.Path} from ClientId {identity.ClientId} has been blocked, quota {rule.Limit}/{rule.Period} exceeded by {counter.TotalRequests}. Blocked by rule { downstreamReRoute.UpstreamPathTemplate.OriginalValue }, TraceIdentifier {httpContext.TraceIdentifier}.");
}
public virtual Task ReturnQuotaExceededResponse(HttpContext httpContext, RateLimitOptions option, string retryAfter)
{
var message = string.IsNullOrEmpty(option.QuotaExceededMessage) ? $"API calls quota exceeded! maximum admitted {option.RateLimitRule.Limit} per {option.RateLimitRule.Period}." : option.QuotaExceededMessage;
if (!option.DisableRateLimitHeaders)
{
httpContext.Response.Headers["Retry-After"] = retryAfter;
}
httpContext.Response.StatusCode = option.HttpStatusCode;
return httpContext.Response.WriteAsync(message);
}
private Task SetRateLimitHeaders(object rateLimitHeaders)
{
var headers = (RateLimitHeaders)rateLimitHeaders;
headers.Context.Response.Headers["X-Rate-Limit-Limit"] = headers.Limit;
headers.Context.Response.Headers["X-Rate-Limit-Remaining"] = headers.Remaining;
headers.Context.Response.Headers["X-Rate-Limit-Reset"] = headers.Reset;
return Task.CompletedTask;
}
}
通過源碼解析,發現實現一個限流還是很簡單的嗎!再進一步解析,IRateLimitCounterHandler
ClientRateLimitProcessor里的相關介面
又是怎麼實現的呢?這時候我們就需要瞭解下.NETCORE 的運行原理,其中ConfigureServices
方法實現了依賴註入(DI)的配置。這時候我們看下Ocelot
是在哪裡進行註入的呢?
services.AddOcelot()
是不是印象深刻呢?原來所有的註入信息都寫在這裡,那麼問題簡單了,Ctrl+F
查找AddOcelot
方法,馬上就能定位到ServiceCollectionExtensions
方法,然後再轉到定義OcelotBuilder
public static class ServiceCollectionExtensions
{
public static IOcelotBuilder AddOcelot(this IServiceCollection services)
{
var service = services.First(x => x.ServiceType == typeof(IConfiguration));
var configuration = (IConfiguration)service.ImplementationInstance;
return new OcelotBuilder(services, configuration);
}
public static IOcelotBuilder AddOcelot(this IServiceCollection services, IConfiguration configuration)
{
return new OcelotBuilder(services, configuration);
}
}
又摸到大動脈啦,現在問題迎刃而解,原來所有的註入都寫在這裡,從這裡可以找下我們熟悉的幾個介面註入。
public OcelotBuilder(IServiceCollection services, IConfiguration configurationRoot)
{
Configuration = configurationRoot;
Services = services;
Services.Configure<FileConfiguration>(configurationRoot);
Services.TryAddSingleton<IOcelotCache<FileConfiguration>, InMemoryCache<FileConfiguration>>();
Services.TryAddSingleton<IOcelotCache<CachedResponse>, InMemoryCache<CachedResponse>>();
Services.TryAddSingleton<IHttpResponseHeaderReplacer, HttpResponseHeaderReplacer>();
Services.TryAddSingleton<IHttpContextRequestHeaderReplacer, HttpContextRequestHeaderReplacer>();
Services.TryAddSingleton<IHeaderFindAndReplaceCreator, HeaderFindAndReplaceCreator>();
Services.TryAddSingleton<IInternalConfigurationCreator, FileInternalConfigurationCreator>();
Services.TryAddSingleton<IInternalConfigurationRepository, InMemoryInternalConfigurationRepository>();
Services.TryAddSingleton<IConfigurationValidator, FileConfigurationFluentValidator>();
Services.TryAddSingleton<HostAndPortValidator>();
Services.TryAddSingleton<IReRoutesCreator, ReRoutesCreator>();
Services.TryAddSingleton<IAggregatesCreator, AggregatesCreator>();
Services.TryAddSingleton<IReRouteKeyCreator, ReRouteKeyCreator>();
Services.TryAddSingleton<IConfigurationCreator, ConfigurationCreator>();
Services.TryAddSingleton<IDynamicsCreator, DynamicsCreator>();
Services.TryAddSingleton<ILoadBalancerOptionsCreator, LoadBalancerOptionsCreator>();
Services.TryAddSingleton<ReRouteFluentValidator>();
Services.TryAddSingleton<FileGlobalConfigurationFluentValidator>();
Services.TryAddSingleton<FileQoSOptionsFluentValidator>();
Services.TryAddSingleton<IClaimsToThingCreator, ClaimsToThingCreator>();
Services.TryAddSingleton<IAuthenticationOptionsCreator, AuthenticationOptionsCreator>();
Services.TryAddSingleton<IUpstreamTemplatePatternCreator, UpstreamTemplatePatternCreator>();
Services.TryAddSingleton<IRequestIdKeyCreator, RequestIdKeyCreator>();
Services.TryAddSingleton<IServiceProviderConfigurationCreator,ServiceProviderConfigurationCreator>();
Services.TryAddSingleton<IQoSOptionsCreator, QoSOptionsCreator>();
Services.TryAddSingleton<IReRouteOptionsCreator, ReRouteOptionsCreator>();
Services.TryAddSingleton<IRateLimitOptionsCreator, RateLimitOptionsCreator>();
Services.TryAddSingleton<IBaseUrlFinder, BaseUrlFinder>();
Services.TryAddSingleton<IRegionCreator, RegionCreator>();
Services.TryAddSingleton<IFileConfigurationRepository, DiskFileConfigurationRepository>();
Services.TryAddSingleton<IFileConfigurationSetter, FileAndInternalConfigurationSetter>();
Services.TryAddSingleton<IServiceDiscoveryProviderFactory, ServiceDiscoveryProviderFactory>();
Services.TryAddSingleton<ILoadBalancerFactory, LoadBalancerFactory>();
Services.TryAddSingleton<ILoadBalancerHouse, LoadBalancerHouse>();
Services.TryAddSingleton<IOcelotLoggerFactory, AspDotNetLoggerFactory>();
Services.TryAddSingleton<IRemoveOutputHeaders, RemoveOutputHeaders>();
Services.TryAddSingleton<IClaimToThingConfigurationParser, ClaimToThingConfigurationParser>();
Services.TryAddSingleton<IClaimsAuthoriser, ClaimsAuthoriser>();
Services.TryAddSingleton<IScopesAuthoriser, ScopesAuthoriser>();
Services.TryAddSingleton<IAddClaimsToRequest, AddClaimsToRequest>();
Services.TryAddSingleton<IAddHeadersToRequest, AddHeadersToRequest>();
Services.TryAddSingleton<IAddQueriesToRequest, AddQueriesToRequest>();
Services.TryAddSingleton<IClaimsParser, ClaimsParser>();
Services.TryAddSingleton<IUrlPathToUrlTemplateMatcher, RegExUrlMatcher>();
Services.TryAddSingleton<IPlaceholderNameAndValueFinder, UrlPathPlaceholderNameAndValueFinder>();
Services.TryAddSingleton<IDownstreamPathPlaceholderReplacer, DownstreamTemplatePathPlaceholderReplacer>();
Services.TryAddSingleton<IDownstreamRouteProvider, DownstreamRouteFinder>();
Services.TryAddSingleton<IDownstreamRouteProvider, DownstreamRouteCreator>();
Services.TryAddSingleton<IDownstreamRouteProviderFactory, DownstreamRouteProviderFactory>();
Services.TryAddSingleton<IHttpRequester, HttpClientHttpRequester>();
Services.TryAddSingleton<IHttpResponder, HttpContextResponder>();
Services.TryAddSingleton<IErrorsToHttpStatusCodeMapper, ErrorsToHttpStatusCodeMapper>();
Services.TryAddSingleton<IRateLimitCounterHandler, MemoryCacheRateLimitCounterHandler>();
Services.TryAddSingleton<IHttpClientCache, MemoryHttpClientCache>();
Services.TryAddSingleton<IRequestMapper, RequestMapper>();
Services.TryAddSingleton<IHttpHandlerOptionsCreator, HttpHandlerOptionsCreator>();
Services.TryAddSingleton<IDownstreamAddressesCreator, DownstreamAddressesCreator>();
Services.TryAddSingleton<IDelegatingHandlerHandlerFactory, DelegatingHandlerHandlerFactory>();
Services.TryAddSingleton<IHttpRequester, HttpClientHttpRequester>();
// see this for why we register this as singleton http://stackoverflow.com/questions/37371264/invalidoperationexception-unable-to-resolve-service-for-type-microsoft-aspnetc
// could maybe use a scoped data repository
Services.TryAddSingleton<IHttpContextAccessor, HttpContextAccessor>();
Services.TryAddSingleton<IRequestScopedDataRepository, HttpDataRepository>();
Services.AddMemoryCache();
Services.TryAddSingleton<OcelotDiagnosticListener>();
Services.TryAddSingleton<IMultiplexer, Multiplexer>();
Services.TryAddSingleton<IResponseAggregator, SimpleJsonResponseAggregator>();
Services.TryAddSingleton<ITracingHandlerFactory, TracingHandlerFactory>();
Services.TryAddSingleton<IFileConfigurationPollerOptions, InMemoryFileConfigurationPollerOptions>();
Services.TryAddSingleton<IAddHeadersToResponse, AddHeadersToResponse>();
Services.TryAddSingleton<IPlaceholders, Placeholders>();
Services.TryAddSingleton<IResponseAggregatorFactory, InMemoryResponseAggregatorFactory>();
Services.TryAddSingleton<IDefinedAggregatorProvider, ServiceLocatorDefinedAggregatorProvider>();
Services.TryAddSingleton<IDownstreamRequestCreator, DownstreamRequestCreator>();
Services.TryAddSingleton<IFrameworkDescription, FrameworkDescription>();
Services.TryAddSingleton<IQoSFactory, QoSFactory>();
Services.TryAddSingleton<IExceptionToErrorMapper, HttpExeptionToErrorMapper>();
//add security
this.AddSecurity();
//add asp.net services..
var assembly = typeof(FileConfigurationController).GetTypeInfo().Assembly;
Services.AddMvcCore()
.AddApplicationPart(assembly)
.AddControllersAsServices()
.AddAuthorization()
.AddJsonFormatters();
Services.AddLogging();
Services.AddMiddlewareAnalysis();
Services.AddWebEncoders();
}
至此Ocelot
源碼解析就到這裡了,其他的具體實現代碼就根據流程一個一個查看即可,這裡就不詳細講解了,因為我們已經掌握整個Ocelot代碼的運行原理和實現方式及流程,項目里其他的一大堆的代碼都是圍繞這個流程去一步一步實現的。
有沒有感覺添加一個中間件不是很複雜呢,是不是都躍躍欲試,準備嘗試開發自己的自定義中間件啦,本篇就不介紹中間件的具體開發流程了,後續實戰中會包含部分項目中需要用到的中間件,到時候會詳細講解如何規劃和開發一個滿足自己項目需求的中間件。
二、結合項目梳理功能
在完整學習完Ocelot文檔和源碼後,我們基本掌握了Ocelot目前已經實現的功能,再結合我們實際項目需求,我們梳理下還有哪些功能可能需要自己擴展實現。
項目設計網關基本需求包括路由、認證、授權、限流、緩存,仔細學習文檔和源碼後發現功能都已經存在,那是不是我們就可以直接拿來使用呢?這時候我們需要拿出一些複雜業務場景來對號入座,看能否實現複雜場景的一些應用。
1、授權
能否為每一個客戶端設置獨立的訪問許可權,如果客戶端A可以訪問服務A、服務B,客戶端B只能訪問服務A,從網關層面直接授權,不滿足需求不路由到具體服務。從文檔和代碼分析後發現暫時未實現。
2、限流
能否為每一個客戶端設置不能限流規則,例如客戶端A為我們內容應用,我希望對服務A不啟用限流,客戶端B為第三方接入應用,我需要B訪問服務A訪問進行單獨限流(30次/分鐘),看能否通過配置實現自定義限流。從文檔和代碼分析後發現暫時未實現。
3、緩存
通過代碼發現目前緩存實現的只是Dictionary方式實現的緩存,不能實現分散式結構的應用。
通過分析我們發現列舉的5個基本需求,盡然有3個在我們實際項目應用中可能會存在問題,如果不解決這些問題,很難直接拿這個完美的網關項目應用到正式項目,所以我們到通過擴展Ocelot方法來實現我們的目的。
如何擴展呢
為了滿足我們項目應用的需要,我們需要為每一個路由進行單獨設置,如果還採用配置文件的方式,肯定無法滿足需求,且後續網關動態增加路由、授權、限流等無法控制,所以我們需要把網關配置信息從配置文件中移到資料庫中,由資料庫中的路由表、限流表、授權表等方式記錄當前網關的應用,且後續擴展直接在資料庫中增加或減少相關配置,然後動態更新網關配置實現網關的高可用。
想一想是不是有點小激動,原來只要稍微改造下寶駿瞬間變寶馬,那接下來的課程就是網關改造之旅,我會從設計、思想、編碼等方面講解下如何實現我們的第一輛寶馬。
本系列文章我也是邊想邊寫邊實現,如果發現中間有任何描述或實現不當的地方,也請各位大神批評指正,我會第一時間整理並修正,避免讓後續學習的人走彎路。