fluentd 설치 (CentOS 7)

공식 사이트

https://www.fluentd.org/

Fluentd | Open Source Data Collector

 

 

Supported Platforms

 

Treasure Data, Inc. maintains stable packages for Fluentd and canonical plugins as Treasure Agent (the package is called td-agent). td-agent has v2 and v3. td-agent v2 for the production and v3 is the new stable version for working with ruby 2.4 and fluetnd v1 series.

Platform	v2	v3
RedHat/CentOS 5	OK
RedHat/CentOS 6/7	OK	OK
Ubuntu Precise	OK
Ubuntu Trusty	OK	OK
Ubuntu Xenial	OK	OK
Ubuntu Bionic	OK	OK
Debian Wheezy	OK
Debian Jessie	OK	OK
Debian Stretch	OK	OK
MacOSX	OK	OK
Windows		OK

Features

 

Major feature updates to td-agent v3 are as follows.

• Ruby 2.4

• Fluentd v1

• Updated for the core libraries, msgpack, Cool.io, etc.

• Windows support

• Drop older distributions and non-popular plugins

• Remove fluentd-ui. This will be released as separated td-agent-ui package

td-agent v3 is now stable version

 

Fluentd v1 is the new stable version.

 

설치

1. NTP 설정

 

AWS 이용자는 chrony 사용

 

• ntp 삭제

sudo yum erase 'ntp*'

• chrony 설치

sudo yum install chrony

• /etc/chrony.conf 파일에 적용

server 169.254.169.123 prefer iburst minpoll 4 maxpoll 4

• chrony 데몬 재시작

sudo service chronyd restart

• 시스템 재시작 시 chrony 적용 설정

sudo chkconfig chronyd on

• chrony 적용 확인

chronyc sources -v

예시

210 Number of sources = 7

  .-- Source mode  '^' = server, '=' = peer, '#' = local clock.
 / .- Source state '*' = current synced, '+' = combined , '-' = not combined,
| /   '?' = unreachable, 'x' = time may be in error, '~' = time too variable.
||                                                 .- xxxx [ yyyy ] +/- zzzz
||      Reachability register (octal) -.           |  xxxx = adjusted offset,
||      Log2(Polling interval) --.      |          |  yyyy = measured offset,
||                                \     |          |  zzzz = estimated error.
||                                 |    |           \
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
^* 169.254.169.123               3   6    17    43    -30us[ -226us] +/-  287us
^- ec2-12-34-231-12.eu-west>     2   6    17    43   -388us[ -388us] +/-   11ms
^- tshirt.heanet.ie              1   6    17    44   +178us[  +25us] +/- 1959us
^? tbag.heanet.ie                0   6     0     -     +0ns[   +0ns] +/-    0ns
^? bray.walcz.net                0   6     0     -     +0ns[   +0ns] +/-    0ns
^? 2a05:d018:c43:e312:ce77:>     0   6     0     -     +0ns[   +0ns] +/-    0ns
^? 2a05:d018:dab:2701:b70:b>     0   6     0     -     +0ns[   +0ns] +/-    0ns

 

참조 : https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html

 

2. Nework Kernel Parameter 최적화

/etc/sysctl.conf에 추가하거나 수정 필요

net.core.somaxconn = 1024
net.core.netdev_max_backlog = 5000
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_wmem = 4096 12582912 16777216
net.ipv4.tcp_rmem = 4096 12582912 16777216
net.ipv4.tcp_max_syn_backlog = 8096
net.ipv4.tcp_slow_start_after_idle = 0
net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_local_port_range = 10240 65535

수정 후 sysctl -p 또는 재부팅

 

3. rpm 이용하여 fluentd 설치

$ curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent3.sh | sh

AWS 적용 시 : 

# Amazon Linux 2
$ curl -L https://toolbelt.treasuredata.com/sh/install-amazon2-td-agent3.sh | sh

 

4. fluentd 시작

$ sudo systemctl start td-agent.service