fluentd 설정 및 도커화 (CentOS 7)

Configuration

컨피그 설정

$ sudo vi /etc/td-agent/td-agent.conf

도커

::: term
docker run -ti --rm -v /path/to/dir:/fluentd/etc fluentd -c /fluentd/etc/<conf-file> -v

 

 

[Host가 Linux or MacOS인 경우] 

1) Host의 localtime을 설정한다 (올바르게 설정되어 있다면 바로 2번으로 넘어가자) 

ln -sf /usr/share/zoneinfo/Asia/Seoul /etc/localtime

2) Host의 localtime 파일을 (ro)읽기전용으로 바인딩하고 Docker Container를 실행한다. 

docker run -v /etc/localtime:/etc/localtime:ro 기타옵션들 

 

 

• 공식 사이트 : https://www.fluentd.org/

 

 

 

 

Supported Platforms

 

Treasure Data, Inc. maintains stable packages for Fluentd and canonical plugins as Treasure Agent (the package is called td-agent). td-agent has v2 and v3. td-agent v2 for the production and v3 is the new stable version for working with ruby 2.4 and fluetnd v1 series.

Platform	v2	v3
RedHat/CentOS 5	OK
RedHat/CentOS 6/7	OK	OK
Ubuntu Precise	OK
Ubuntu Trusty	OK	OK
Ubuntu Xenial	OK	OK
Ubuntu Bionic	OK	OK
Debian Wheezy	OK
Debian Jessie	OK	OK
Debian Stretch	OK	OK
MacOSX	OK	OK
Windows		OK

 

Features

 

Major feature updates to td-agent v3 are as follows.

• Ruby 2.4

• Fluentd v1

• Updated for the core libraries, msgpack, Cool.io, etc.

• Windows support

• Drop older distributions and non-popular plugins

• Remove fluentd-ui. This will be released as separated td-agent-ui package

td-agent v3 is now stable version

 

Fluentd v1 is the new stable version.

 

 

 

설치

1. NTP 설정

• AWS 이용자는 chrony 사용
• ntp 삭제

 

sudo yum erase 'ntp*'

• chrony 설치

 

sudo yum install chrony

• /etc/chrony.conf 파일에 적용

 

server 169.254.169.123 prefer iburst minpoll 4 maxpoll 4

• chrony 데몬 재시작

 

sudo service chronyd restart

• 시스템 재시작 시 chrony 적용 설정

sudo chkconfig chronyd on

• chrony 적용 확인

참조 : https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html

 

2. Nework Kernel Parameter 최적화 ( 서비스 서버에는 적용하면 안됨)

/etc/sysctl.conf에 추가하거나 수정 필요

net.core.somaxconn = 1024

net.core.netdev_max_backlog = 5000

net.core.rmem_max = 16777216

net.core.wmem_max = 16777216

net.ipv4.tcp_wmem = 4096 12582912 16777216

net.ipv4.tcp_rmem = 4096 12582912 16777216

net.ipv4.tcp_max_syn_backlog = 8096

net.ipv4.tcp_slow_start_after_idle = 0

net.ipv4.tcp_tw_reuse = 1

net.ipv4.ip_local_port_range = 10240 65535

 

수정 후 sysctl -p 또는 재부팅

 

3. rpm 이용하여 fluentd 설치

$ curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent3.sh | sh

AWS 적용 시 : 

# Amazon Linux 2 $ curl -L https://toolbelt.treasuredata.com/sh/install-amazon2-td-agent3.sh | sh

 

4. fluentd 시작

$ sudo systemctl start td-agent.service

 

 

<tunning & trouble shooting>

 

• <ip address>. log 읽어 들이지 않기
  
  

<source>   @type tail   path /logdata/access/*.com.log,/logdata/access/*.net.log,/logdata/access/*.kr.log   exclude_path ["/logdata/access/*.gz", "/logdata/access/*.zip" ] </source>   exclude_path : "/logdata/access/*.[[digit:]].log" 추가 -->  IP주소log 걸러내기 > 안됨

 

• Dockerfiles 위치한 폴더 내에 넣을 스크립트 및 conf 파일

<system>

  root_dir /var/log/td-agent         # For handling unrecoverable chunks
</system>
 
<source>
  @type tail
  path /logdata/access/*.com.log,/logdata/access/*.net.log,/logdata/access/*.kr.log
  exclude_path ["/logdata/access/*.gz", "/logdata/access/*.zip" ]
  pos_file /var/log/td-agent/nginx-access.log.pos
  tag nginx.raw
  refresh_interval 1s
  <parse>
    @type tsv
    keys domain,date,time,url,sent,res,ret,status,cip,agent,referer,upip,up_status,geo,geoorg,encoding,length,up1,up2,up3
    time_key time
    types sent:integer, res:float, up1:float, up2:float, up3:float
  </parse>
</source>
 
<filter nginx.raw>
  @type record_modifier
  remove_keys date, encoding, length
  <record>
    host "#{Socket.gethostname}"
  </record>
  <replace>
    key agent
    expression /^$/
    replace -
  </replace>
  <replace>
    key referer
    expression /^$/
    replace -
  </replace>
</filter>
 
<match nginx.*>
  @type forward
  <server>
    host <IP 정보>
    port 24224
  </server>
  <server>
    host <IP 정보>
    port 24224
    standby
  </server>
  heartbeat_type transport
  <buffer>
    flush_interval 1
    retry_wait 1                      # The wait interval for the first retry.
    retry_exponential_backoff_base 2  # Inclease the wait time by a factor of N.
    retry_type exponential_backoff    # Set 'periodic' for constant intervals.
    retry_randomize true              # Apply randomization. (see above)
    retry_timeout 72h                 # Maximum duration before giving up.
    retry_forever false               # Set 'true' for infinite retry loops.
    retry_secondary_threshold 0.8     # See the "Secondary Output" section in
  </buffer>
  <secondary>
     @type file
     path /var/log/td-agent/buffer/nginx.raw
     compress gzip
  </secondary>
</match>

• Docker build

docker build -t caclog:1.05 .

Docker image 받기

docker pull <도커허브도메인 또는 IP>/caclog:1.05

 

Docker 실행 명령어

docker run -d -v /logdata/access/:/logdata/access/ --net=host --name=caclog_1.06 regi.myskcdn.net:5000/caclog:1.06