文件同步傳輸工具比較多,傳輸的方式也比較多,比如:FTP、共用、HTTP等,我這裡要講的就是基於HTTP協議的WEB API實現批量文件由一個伺服器同步快速傳輸到其它多個伺服器這樣的一個工具(簡稱:一端到多端的文件同步工具) 一、設計原理: 1.使用的技術:WinForm、WebApi 1.1 Wi ...
文件同步傳輸工具比較多,傳輸的方式也比較多,比如:FTP、共用、HTTP等,我這裡要講的就是基於HTTP協議的WEB API實現批量文件由一個伺服器同步快速傳輸到其它多個伺服器這樣的一個工具(簡稱:一端到多端的文件同步工具)
一、設計原理:
1.使用的技術:WinForm、WebApi
1.1 WinForm:為程式主界面,作為一端(一個源文件伺服器)同步傳輸到多端(多個目的文件伺服器)文件的業務處理中介;程式內部主要通過System.Timers.Timer+HttpClient來實現定時執行文件同步傳輸業務;
1.2 WebApi:實現通過HTTP協議批量下載或批量上傳多個文件(文件同步傳輸的核心業務邏輯);MultipartContent作為批量下載或批量上傳的唯一媒介。
2.實現思路:
2.1客戶端(WinForm程式主界面)通過HttpClient向源文件伺服器目錄URL發送GET請求;
2.2源文件伺服器服務端(WebApi)的GetFiles方法接收到GET請求後,按照web.config中配置的源文件路徑遞歸獲取所有文件的位元組信息並轉換成MultipartFormDataContent對象後返回;(即:實現了批量下載)
2.3客戶端(WinForm程式主界面)將響應的結果顯式轉換並生成對應的目的文件伺服器數量的多文件流內容(MultipartContent)對象列表,以便後續用於批量上傳;
2.4客戶端(WinForm程式主界面)啟用並行迴圈(Parallel.ForEach)來遍歷目的文件伺服器目錄URL(採用並行迴圈是為了達到同時向多個目的文件伺服器批量上傳文件的效果,從而提高運行效率),在迴圈遍歷中,每次將2.3中獲得的多文件流內容(MultipartContent)通過HttpClient向目的文件伺服器目錄URL發送POST請求;
2.5目的文件伺服器服務端(WebApi)的SaveFiles方法接收到POST請求後,解析2.4中POST過來的多文件流內容(MultipartContent),並迴圈遍歷文件流,在迴圈遍歷中,按照web.config中配置的上傳文件路徑,將文件流輸出保存到指定的文件路徑下,同時生成文件保存成功與失敗的日誌信息字典,最後返回該字典。(即:實現了批量上傳保存)
2.6客戶端(WinForm程式主界面)將響應的結果顯式轉換成保存成功與失敗的日誌信息字典,並添加到線程安全的無序集合對象中;(採用線程安全的無序集合對象是因為存在多線程併發更新的風險)
2.7客戶端(WinForm程式主界面)等待所有並行迴圈同步上傳執行完畢後,根據最後得到的保存成功與失敗的日誌信息無序集合對象,獲得所有目的文件伺服器全部保存成功文件名列表及保存成功與失敗的日誌信息列表(判斷是否全部上傳成功:若某個文件應上傳到5個目的文件伺服器,實際成功上傳5個,則視為成功,否則有一個未上傳成功則視為失敗),然後通過HttpClient向源文件伺服器目錄URL發送PUT請求刪除源文件伺服器中的同名文件,向源文件伺服器LOG URL發送POST請求將此次文件同步傳輸的日誌保存到源文件伺服器目錄中
2.8源文件伺服器服務端(WebApi)RemoveFiles方法接收到PUT請求後,迴圈遍歷PUT過來的保存成功文件名列表,依次刪除同名文件(含子目錄),WriteLog方法接收到POST請求後,直接將POST過來的日誌信息列表輸出保存至源文件伺服器web.config中配置的LOG文件路徑;為了避免日誌無限增大及考慮日誌的使用價值,日誌文件每天重新覆蓋生成新的文件;
3.業務流程圖:
序列圖:
二、使用說明:
1.將KyFileSyncTransfer.Api項目分別發佈到源文件伺服器、目的文件伺服器;
2.修改源文件伺服器服務端(WebApi)web.config中的AppSettings節點,配置FileSyncDirectory(源文件存放目錄)、LogDirectory(日誌文件保存路徑,僅限源文件伺服器服務端配置),這兩個路徑支持當前項目的子目錄(即:配置時使用~/)或其它路徑(其它路徑則需直接配置完整的物理路徑);修改目的文件伺服器服務端(WebApi)web.config中的AppSettings節點,配置FileSyncDirectory(上傳文件存放目錄),這個路徑支持當前項目的子目錄(即:配置時使用~/)或其它路徑(其它路徑則需直接配置完整的物理路徑)
註:為了能夠支持大文件批量上傳,同時需修改請求內容長度限制,如下設置成最大批量上傳1.9G:
3.將客戶端(WinForm程式主界面)部署到某台伺服器上(只要能夠訪問源文件伺服器、目的文件伺服器即可,也可以他們伺服器上的任意一臺),然後開啟客戶端(WinForm程式主界面),將上述的源文件伺服器服務端URL(http://源文件伺服器Host/Api/Files)及多個目的文件伺服器服務端URL(http://目的文件伺服器Host/Api/Files)錄入到程式預設的地方,設置好時間間隔,最後點擊開啟即可(需保持該程式一直處於運行中,可以最小化到任務欄,雙擊圖標可以顯示界面);若需停止文件同步,點擊停止按鈕即可;若需查看運行日誌,可以切換到運行日誌頁簽瀏覽。
4.以上3步是完成了文件自動定時同步傳輸的所有工作,後續只需要將需要同步的文件放到源文件伺服器服務端web.config中的AppSettings節點設置的FileSyncDirectory(源文件存放目錄)即可。
運行效果如下:
三、貼出主要的源代碼
服務端代碼(WEB API代碼,需要進行文件傳輸的每個伺服器均需要部署該WEB API站點)
FilesController:(實現:批量下載文件、批量上傳文件、批量刪除文件、批量寫日誌信息)
using System; using System.Collections.Generic; using System.IO; using System.Linq; using System.Net; using System.Net.Http; using System.Web; using System.Web.Http; using KyFileSyncTransfer.Api.Common; using System.Net.Http.Headers; using System.Net.Http.Formatting; using System.Threading.Tasks; namespace KyFileSyncTransfer.Api.Controllers { public class FilesController : ApiController { private static string fileSyncDirectory = null; private static string logDirectory = null; static FilesController() { fileSyncDirectory = BaseUtility.GetDirectoryFromConfig("FileSyncDirectory"); logDirectory = BaseUtility.GetDirectoryFromConfig("LogDirectory"); } /// <summary> /// 從源文件伺服器獲取所有文件信息(採用JSON方式) /// </summary> /// <returns></returns> [HttpGet, Route("~/api/downfiles")] public IHttpActionResult GetFilesForJson() { if (!Directory.Exists(fileSyncDirectory)) { return BadRequest("同步文件目錄不存在或未配置。"); } Dictionary<string, byte[]> files = new Dictionary<string, byte[]>(); BaseUtility.LoadFileDatas(files, fileSyncDirectory); files = files.ToDictionary(kv => kv.Key.Replace(fileSyncDirectory, ""), kv => kv.Value); return Json(files); } /// <summary> /// 將所有文件同步保存到目的文件伺服器(採用JSON方式) /// </summary> /// <param name="files"></param> /// <returns></returns> [HttpPost, Route("~/api/upfiles")] public IHttpActionResult SaveFilesForJson([FromBody]IDictionary<string, byte[]> files) { string requestUrl = HttpContext.Current.Request.Url.ToString(); var savedErrors = new Dictionary<string, string>(); if (files == null || !Directory.Exists(fileSyncDirectory)) { return BadRequest(); } foreach (var item in files) { string file = item.Key; string filePath = Path.GetDirectoryName(fileSyncDirectory + file); try { if (!Directory.Exists(filePath)) { Directory.CreateDirectory(filePath); } string saveFilePath = Path.Combine(filePath, Path.GetFileName(file)); File.WriteAllBytes(saveFilePath, item.Value); } catch (Exception ex) { savedErrors[file] = string.Format("[{0:yyyy-MM-dd HH:mm:ss}] -請求:{1}同步文件:{2}失敗,原因:{3}", DateTime.Now, requestUrl, file, ex.Message); } } return Json(savedErrors); } /// <summary> /// 從源文件伺服器獲取所有文件信息 /// </summary> /// <returns></returns> [HttpGet] public HttpResponseMessage GetFiles() { if (!Directory.Exists(fileSyncDirectory)) { return Request.CreateErrorResponse(HttpStatusCode.BadRequest, "同步文件目錄不存在或未配置。"); } var response = new HttpResponseMessage(HttpStatusCode.OK); var content = new MultipartFormDataContent(); BaseUtility.CreateMultipartFormDataContent(content, fileSyncDirectory, fileSyncDirectory); response.Content = content; return response; } /// <summary> /// 將所有文件同步保存到目的文件伺服器 /// </summary> /// <returns></returns> [HttpPost] public HttpResponseMessage SaveFiles() { if (!Request.Content.IsMimeMultipartContent()) { return Request.CreateErrorResponse(HttpStatusCode.UnsupportedMediaType, "未上傳任何文件"); } if (!Directory.Exists(fileSyncDirectory)) { return Request.CreateErrorResponse(HttpStatusCode.BadRequest, "未找到文件同步上傳目錄:" + fileSyncDirectory); } string requestUrl = HttpContext.Current.Request.Url.ToString(); Dictionary<string, Dictionary<string, string>> savedResult = new Dictionary<string, Dictionary<string, string>>(); var provider = new MultipartMemoryStreamProvider(); const string success = "success"; const string failure = "failure"; try { savedResult[success] = new Dictionary<string, string>(); savedResult[failure] = new Dictionary<string, string>(); //Request.Content.ReadAsMultipartAsync(provider).Wait(); Task.Run(async () => await Request.Content.ReadAsMultipartAsync(provider)).Wait(); foreach (var item in provider.Contents) { string fileName = item.Headers.ContentDisposition.FileName; if (string.IsNullOrEmpty(fileName)) { continue; } var fileData = item.ReadAsByteArrayAsync().Result; fileName = BaseUtility.ReviseFileName(fileName); string saveFilePath = fileSyncDirectory + fileName; string fileBasePath = Path.GetDirectoryName(saveFilePath); try { if (!Directory.Exists(fileBasePath)) { Directory.CreateDirectory(fileBasePath); } File.WriteAllBytes(saveFilePath, fileData); savedResult[success][fileName] = string.Format("[{0:yyyy-MM-dd HH:mm:ss}] - V 請求:{1}同步文件:<{2}>成功。", DateTime.Now, requestUrl, fileName); } catch (Exception ex) { while(ex.InnerException!=null) { ex = ex.InnerException; } savedResult[failure][fileName] = string.Format("[{0:yyyy-MM-dd HH:mm:ss}] - X 請求:{1}同步文件:<{2}>失敗,原因:{3}", DateTime.Now, requestUrl, fileName, ex.Message); } } } catch (Exception ex) { while (ex.InnerException != null) { ex = ex.InnerException; } return Request.CreateErrorResponse(HttpStatusCode.BadRequest, ex.Message); } return Request.CreateResponse(HttpStatusCode.OK, savedResult); } /// <summary> /// 移除源文件伺服器指定的文件 /// </summary> /// <param name="files"></param> /// <returns></returns> [HttpPut] public IHttpActionResult RemoveFiles([FromBody]IEnumerable<string> files) { if (files == null || !Directory.Exists(fileSyncDirectory)) { return BadRequest(); } foreach (string file in files) { string filePath = Path.Combine(fileSyncDirectory, file); if (File.Exists(filePath)) { File.Delete(filePath); } } return Ok(); } /// <summary> /// 將同步的日誌信息寫入到源文件伺服器LOG文件中 /// </summary> /// <param name="savedErrors"></param> /// <returns></returns> [HttpPost] [Route("~/Api/Files/Log")] public IHttpActionResult WriteLog([FromBody] IEnumerable<string> savedResult) { if (!Directory.Exists(logDirectory)) { return BadRequest("同步日誌目錄不存在或未配置。"); } BaseUtility.WriteLogToFile(logDirectory, savedResult.ToArray()); return Ok(); } } }
WebApiConfig:(增加全局token驗證及全部採用json數據返回)
using System; using System.Collections.Generic; using System.Linq; using System.Web.Http; using KyFileSyncTransfer.Api.Models; using System.Net.Http.Formatting; namespace FileSyncTransfer.Api { public static class WebApiConfig { public static void Register(HttpConfiguration config) { // Web API 配置和服務 // Web API 路由 config.MapHttpAttributeRoutes(); config.Routes.MapHttpRoute( name: "DefaultApi", routeTemplate: "api/{controller}/{id}", defaults: new { id = RouteParameter.Optional } ); config.Filters.Add(new TokenAuthentificationAttribute()); var jsonFormatter = new JsonMediaTypeFormatter(); config.Services.Replace(typeof(IContentNegotiator), new JsonContentNegotiator(jsonFormatter)); } } }
JsonContentNegotiator:
public class JsonContentNegotiator : IContentNegotiator { private readonly JsonMediaTypeFormatter _jsonFormatter; public JsonContentNegotiator(JsonMediaTypeFormatter formatter) { _jsonFormatter = formatter; } public ContentNegotiationResult Negotiate(Type type, HttpRequestMessage request, IEnumerable<MediaTypeFormatter> formatters) { var result = new ContentNegotiationResult(_jsonFormatter, new MediaTypeHeaderValue("application/json")); return result; } }
TokenAuthentificationAttribute:
public class TokenAuthentificationAttribute : AuthorizationFilterAttribute { public override void OnAuthorization(System.Web.Http.Controllers.HttpActionContext actionContext) { if (actionContext.ActionDescriptor.GetCustomAttributes<AllowAnonymousAttribute>().Count > 0) { base.OnAuthorization(actionContext); return; } //HttpContextBase context = (HttpContextBase)actionContext.Request.Properties["MS_HttpContext"];//獲取傳統context //HttpRequestBase request = context.Request;//定義傳統request對象 IEnumerable<string> requestToken = null; if (actionContext.Request.Headers.TryGetValues("token", out requestToken) && BaseUtility.ValidateToken(requestToken.ElementAt(0))) { base.OnAuthorization(actionContext); } else { actionContext.Response = actionContext.Request.CreateErrorResponse(HttpStatusCode.Unauthorized, "token驗證未通過。"); return; } } }
BaseUtility:(通用方法類)
using System; using System.Collections.Generic; using System.Linq; using System.Web; using System.Configuration; using System.Net.Http; using System.IO; using System.Net.Http.Headers; namespace KyFileSyncTransfer.Api.Common { public static class BaseUtility { public static string GetDirectoryFromConfig(string cfgName) { string dir = ConfigurationManager.AppSettings[cfgName]; if (string.IsNullOrEmpty(dir)) { return null; } if (dir.Contains('~')) { dir = HttpContext.Current.Server.MapPath(dir); } if (!Directory.Exists(dir)) { Directory.CreateDirectory(dir); } return dir; } public static MultipartFormDataContent CreateMultipartFormDataContent(MultipartFormDataContent content, string removeRootDir, string dir) { foreach (string file in Directory.GetFileSystemEntries(dir)) { if (File.Exists(file)) { byte[] fileBytes = File.ReadAllBytes(file); var fileContent = new ByteArrayContent(fileBytes); fileContent.Headers.ContentDisposition = new ContentDispositionHeaderValue("form-data") { Name = "files", FileName = file.Replace(removeRootDir, "") }; fileContent.Headers.ContentType = new MediaTypeHeaderValue(MimeMapping.GetMimeMapping(file)); fileContent.Headers.ContentLength = fileBytes.LongLength; content.Add(fileContent); } else { CreateMultipartFormDataContent(content, removeRootDir, file); } } return content; } public static void LoadFileDatas(Dictionary<string, byte[]> files, string path) { foreach (string file in Directory.GetFileSystemEntries(path)) { if (File.Exists(file)) { files[file] = File.ReadAllBytes(file); } else { LoadFileDatas(files, file); } } } public static void WriteLogToFile(string logDir, params string[] contents) { string logFilePath = Path.Combine(logDir, "KyFileSyncTransfer.log"); if (File.Exists(logFilePath) && !File.GetLastWriteTime(logFilePath).Date.Equals(DateTime.Today)) { File.Delete(logFilePath); } File.AppendAllLines(logFilePath, contents, System.Text.Encoding.UTF8); } public static bool ValidateToken(string token) { try { token = EncryptUtility.Decrypt(token); var tokenParts = token.Split(new[] { "-", string.Empty }, StringSplitOptions.RemoveEmptyEntries); if (tokenParts.Length != 2) { return false; } if (tokenParts[0] == string.Join(string.Empty, "KyFileSyncTransfer.Api".OrderBy(c => c))) //對固定KEY進行排序然後比對 { long tokenTstamp = -1; long svrTokenTimeStamp = GetTimeStamp(); if (long.TryParse(tokenParts[1], out tokenTstamp) && svrTokenTimeStamp - tokenTstamp <= 10) //時間戳<=10則視為有效 { return true; } } } catch { } return false; } public static string ReviseFileName(string fileName) { var regex = new System.Text.RegularExpressions.Regex("^\"+(?<name>.*)\"+$"); var matchResult = regex.Match(fileName); if (matchResult != null && matchResult.Length > 0) { return matchResult.Groups["name"].Value; } return fileName; } /// <summary> /// 獲取時間戳 /// </summary> /// <returns></returns> private static long GetTimeStamp() { TimeSpan ts = DateTime.UtcNow - new DateTime(1970, 1, 1, 0, 0, 0, 0); return Convert.ToInt64(ts.TotalSeconds); } } }
客戶端代碼:(這裡面有一個需要註意的地方就是:GetNeedSyncTransferFilesDatas方法,這個是將從源文件伺服器下載有流轉換成多個副本的多文件流對象,之前是用的GetNeedSyncTransferFilesData方法,但MultipartContent是一種位元組流對象,一旦被用於請求後將會被關閉,再次使用時就會報錯)
using System; using System.Collections.Generic; using System.ComponentModel; using System.Data; using System.Drawing; using System.Linq; using System.Text; using System.Threading.Tasks; using System.Windows.Forms; using System.Threading; using KyFileSyncTransfer.Business; using System.Diagnostics; using System.Net.Http; using System.Collections.Concurrent; using System.IO; using System.Runtime.Serialization.Formatters.Binary; using System.Net.Http.Headers; namespace KyFileSyncTransfer { public partial class FrmMain : Form { private const string appVersion = "16.1215.1"; private FormWindowState thisFormWindowState; private System.Timers.Timer appTimer = null; private static int syncFlag = 0; private static object syncObj = new object(); private static DateTime lastRunTime = DateTime.MinValue; private int runInterval = 10; private string srcFileApiUrl = null; private List<WebApiUrlInfo> destFileApiUrlList = null; private const string success = "success"; private const string failure = "failure"; private const string RunInterval = "RunInterval"; private const string SrcFileApiUrl = "SrcFileApiUrl"; private const string DestFileApiUrls = "DestFileApiUrls"; public FrmMain() { InitializeComponent(); } #region 自定義方法區域 private void ExecuteFileTransfer() { List<string> srcFileNameList = new List<string>(); var needSyncTransferFilesDatas = GetNeedSyncTransferFilesDatas(srcFileNameList, destFileApiUrlList.Count).Result; WriteLog(string.Format("從源伺服器目錄Http Url:{0},獲取到{1}個需要同步的文件.", srcFileApiUrl, srcFileNameList.Count)); if (needSyncTransferFilesDatas == null || srcFileNameList.Count <= 0) return; ShowFileInfoLogs("需要同步的文件列表如下:", srcFileNameList); var fileTransferResultBag = new ConcurrentBag<Dictionary<string, Dictionary<string, string>>>(); Parallel.ForEach(destFileApiUrlList, (destFileApiUrl) => { MultipartContent needSyncTransferFilesData = null; if (needSyncTransferFilesDatas.TryTake(out needSyncTransferFilesData)) { var savedResult = new Dictionary<string, Dictionary<string, string>>(); try { savedResult = SyncTransferFiles(destFileApiUrl.Url, needSyncTransferFilesData).Result; } catch (Exception ex) { while (ex.InnerException != null) { ex = ex.InnerException; } savedResult[success] = new Dictionary<string, string>(); savedResult[failure] = new Dictionary<string, string>(); savedResult[failure][destFileApiUrl.Url] = string.Format("[{0:yyyy-MM-dd HH:mm:ss}] - X 請求:{1} 響應失敗,原因:{3}", DateTime.Now, destFileApiUrl, ex.Message); } fileTransferResultBag.Add(savedResult); ShowSyncTransferFileLogs(savedResult); } }); #region 同步迴圈 //foreach (var destFileApiUrl in destFileApiUrlList) //{ // MultipartContent needSyncTransferFilesData = null; // if (needSyncTransferFilesDatas.TryTake(out needSyncTransferFilesData)) // { // var savedResult = new Dictionary<string, Dictionary<string, string>>(); // try // { // savedResult = SyncTransferFiles(destFileApiUrl.Url, needSyncTransferFilesData).Result; // } // catch (Exception ex) // { // while (ex.InnerException != null) // { // ex = ex.InnerException; // } // savedResult[success] = new Dictionary<string, string>(); // savedResult[failure] = new Dictionary<string, string>(); // savedResult[failure][destFileApiUrl.Url] = string.Format("[{0:yyyy-MM-dd HH:mm:ss}] - X 請求:{1} 響應失敗,原因:{2}", DateTime.Now, destFileApiUrl, ex.Message); // } // fileTransferResultBag.Add(savedResult); // ShowSyncTransferFileLogs(savedResult); // } //} #endregion List<string> needRemoveFileNameList = GetNeedRemoveFileNameList(srcFileNameList, fileTransferResultBag.Select(b => b[success])); RemoveSourceFiles(needRemoveFileNameList); WriteSyncTransferFileLog(GetSyncTransferFileLogList(fileTransferResultBag)); ShowFileInfoLogs("以下文件已成功同步保存到預設的所有目的伺服器目錄Http Url中,且已移除在源伺服器目錄Http Url中的相同文件:", needRemoveFileNameList); } private void ShowFileInfoLogs(string logTitle, List<string> fileList) { WriteLog(logTitle); foreach (string file in fileList) { WriteLog(file); } WriteLog("-".PadRight(30, '-')); } private void ShowSyncTransferFileLogs(Dictionary<string, Dictionary<string, string>> savedResult) { foreach (var kv in savedResult) { bool isError = (kv.Key == failure); foreach (var kv2 in kv.Value) { WriteLog(kv2.Value, isError); } } } private void ReviseFileNames(List<string> fileNameList) { var regex = new System.Text.RegularExpressions.Regex("^\"+(?<name>.*)\"+$"); for (int i = 0; i < fileNameList.Count; i++) { string fileName = fileNameList[i]; var matchResult = regex.Match(fileName); if (matchResul