ELK系列~Nxlog日誌收集加轉發(解決log4日誌換行導致json轉換失敗問題)

本文章將會繼承上一篇文章，主要講通過工具來進行日誌的收集與發送，《ELK系列~NLog.Targets.Fluentd到達如何通過tcp發到fluentd》

Nxlog是一個日誌收集工具，它將系統日誌，或者指定的日誌文件，統配符文件找到，然後加工，最後發送到目標位置。而目標位置有很多種，如文件系統，fluentd系統等，下麵我們介紹一個使用場景，也是經常涉及到的場景。

1. log4產生日期，日期文件名，統一尾碼，按日誌級別命名
2. nxlog工具，配置，啟動，發送數據
3. fluentd配置，接受數據，列印數據

一 log4產生日期，日期文件名，統一尾碼，按日誌級別命名

<?xml version="1.0"?>
<configuration>
  <configSections>
      <section name="log4net" type="log4net.Config.Log4NetConfigurationSectionHandler,log4net" />
  </configSections>
  <system.web>
    <compilation debug="true" targetFramework="4.0" />
  </system.web>

  <log4net>
    <logger name="ErrorLog">
      <level value="ERROR"></level>
      <appender-ref ref="ErrorLog"></appender-ref>
    </logger>
    <appender name="ErrorLog" type="log4net.Appender.RollingFileAppender">
      <param name="File" value="logs\\" />
      <param name="AppendToFile" value="true" />
      <param name="StaticLogFileName" value="false" />
      <rollingStyle value="Date" />
      <datePattern value="yyyyMMdd&quot;Error.log&quot;" />
      <layout type="log4net.Layout.PatternLayout">
      </layout>
    </appender>

    <logger name="InfoLog">
      <level value="INFO"></level>
      <appender-ref ref="InfoLog"></appender-ref>
    </logger>
    <appender name="InfoLog" type="log4net.Appender.RollingFileAppender">
      <param name="File" value="logs\\" />
      <param name="AppendToFile" value="true" />
      <param name="StaticLogFileName" value="false" />
      <rollingStyle value="Date" />
      <datePattern value="yyyyMMdd&quot;Info.log&quot;" />
      <layout type="log4net.Layout.PatternLayout">
      </layout>
    </appender>

    <logger name="WarnLog">
      <level value="Warn"></level>
      <appender-ref ref="WarnLog"></appender-ref>
    </logger>
    <appender name="WarnLog" type="log4net.Appender.RollingFileAppender">
      <param name="File" value="logs\\" />
      <param name="AppendToFile" value="true" />
      <param name="StaticLogFileName" value="false" />
      <rollingStyle value="Date" />
      <datePattern value="yyyyMMdd&quot;Warn.log&quot;" />
      <layout type="log4net.Layout.PatternLayout">
      </layout>
    </appender>

    <logger name="FatalLog">
      <level value="Fatal"></level>
      <appender-ref ref="FatalLog"></appender-ref>
    </logger>
    <appender name="FatalLog" type="log4net.Appender.RollingFileAppender">
      <param name="File" value="logs\\" />
      <param name="AppendToFile" value="true" />
      <param name="StaticLogFileName" value="false" />
      <rollingStyle value="Date" />
      <datePattern value="yyyyMMdd&quot;Fatal.log&quot;" />
      <layout type="log4net.Layout.PatternLayout">
      </layout>
    </appender>
  </log4net>
</configuration>

產生日誌的方法如下

            log4net.Config.XmlConfigurator.Configure(new FileInfo("log4.config"));
            for (int i = 0; i < 10; i++)
            {
                LogHelper.Info("test");
            }
            Console.WriteLine("end");

上面代碼將會產生日誌文件，命名比較固定，方法日誌的收集工作。

二 nxlog工具，配置，啟動，發送數據

安裝：https://docs.fluentd.org/v0.12/articles/windows

配置：去掉了log4產生日誌每行後面的\r標記，如果不進行處理，在json轉換時會有問題

## This is a sample configuration file. See the nxlog reference manual about the
## configuration options. It should be installed locally and is also available
## online at http://nxlog.org/docs/

## Please set the ROOT to the folder your nxlog was installed into,
## otherwise it will not start.

#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log


 <Extension json>
    Module      xm_json
</Extension>
<Extension _syslog>
    Module      xm_syslog
</Extension>

<Input in>
    Module    im_file
    File    "c:\dotnet\20*.log"
    Exec $raw_event = replace($raw_event, "\r\n", " ");
    Exec $raw_event = replace($raw_event, "\r", " ");
    Exec $raw_event = replace($raw_event, "\n", " ");
    Exec $raw_event = replace($raw_event, "0x0A", " ");
    Exec $raw_event = replace($raw_event, "0x0DA", " ");
    Exec $raw_event = replace($raw_event, "0x0D", " ");

 </Input>
<Output out>
    Module      om_tcp
    Host        192.168.200.214
    Port        24224
    Exec $raw_event =$raw_event + "\n";
</Output>

<Route 1> 
    Path        in => out
</Route>

啟動：nxlog -f -c conf/nxlog.conf

查看：打開fluentd端，查看它的日誌，發現我們的日誌已經過來了

這個東西，我在google上找了好久，最後總算是功夫不負有心人，讓我找對了！當然也證明瞭最初的猜想是正確的，即在output中對字元進行過濾！

三 fluentd配置，接受數據，列印數據

1 運行腳本，升級需要的文件夾和文件

mkdir /scripts/fluentd -p
cd /scripts/fluentd

cat  > Dockerfile << 'EOF'
FROM fluent/fluentd:v0.12-onbuild

ENV TZ=Asia/Shanghai
RUN echo "http://mirrors.aliyun.com/alpine/v3.5/main" >/etc/apk/repositories \
        && echo "http://mirrors.aliyun.com/alpine/v3.5/community" >>/etc/apk/repositories \
        && apk add --update tzdata \
        && apk add curl \
        && ln -snf /usr/share/zoneinfo/$TZ /etc/localtime \
        && echo $TZ > /etc/timezone

RUN apk add  --virtual .build-deps \
        sudo build-base ruby-dev \
&& sudo gem sources --clear-all \
&& apk del .build-deps \
&& rm -rf /var/cache/apk/* \
 /home/fluent/.gem/ruby/2.3.0/cache/*.gem
EOF

mkdir plugins -p
mkdir -p /srv/volume/fluentd/

cat > /scripts/fluentd/fluent.conf << 'EOF'
<source>
  @type forward
  port  24224
  bind 0.0.0.0
</source>

<match **>
  @type stdout
</match>
EOF

2 生成鏡像

docker build --no-cache --pull -t pilipa/tools/fluentd ./

2 docker上直接運行

docker run --privileged=true -v /scripts/fluentd/fluent.conf:/fluentd/etc/fluent.conf pilipa/tools/fluentd