😀
virtualbox(centos)でのfluentdインストールメモ(apacheのアクセスログ収集まで)
環境 centos6.3(ゲストOS)macosx10.8(ホストOS)
- ruby1.9.1以上gem1.8.1以上が入ってる前提とする
- httpd(apache)がインストール済とする
fluentdとは
ログなどをjson化できるもので日本人の古橋さんが開発したものです。
有名企業でも導入がされています。
- slideshare
- GREE 等
公式ドキュメント
日本語版(非公式??)開発でよくある「アクセスログを解析してくれーーー」って時に
jsonでオブジェクト化されたデータの精査ができるので便利そうです。
(まだ使い始めなのでよくわからんが。。。。)
gemのupdateをする
一番はじめ公式ドキュメント通りやろうと思ったらエラーになってしばし悩んだ。。。
gem install fluentd --no-ri --no-rdoc
WARNING: Error fetching data: SocketError: getaddrinfo: Name or service not known (http://rubygems.org/latest_specs.4.8.gz)
ERROR: Could not find a valid gem 'fluentd' (>= 0) in any repository
ERROR: Possible alternatives: fluentd
悩むこと1時間まずはupdateかと気づく
なんかの依存モジュールのパッケージで引っ掛かんだと思う。
gem update
Updating installed gems
WARNING: Error fetching data: SocketError: getaddrinfo: Name or service not known (http://rubygems.org/latest_specs.4.8.gz)
Updating io-console
WARNING: Error fetching data: SocketError: getaddrinfo: Name or service not known (http://rubygems.org/specs.4.8.gz)
Fetching: io-console-0.4.1.gem (100%)
Building native extensions. This could take a while...
Successfully installed io-console-0.4.1
Updating minitest
Fetching: minitest-4.5.0.gem (100%)
Successfully installed minitest-4.5.0
Updating rake
Fetching: rake-10.0.3.gem (100%)
Successfully installed rake-10.0.3
Updating rdoc
Fetching: rdoc-3.12.1.gem (100%)
Depending on your version of ruby, you may need to install ruby rdoc/ri data:
<= 1.8.6 : unsupported
= 1.8.7 : gem install rdoc-data; rdoc-data --install
= 1.9.1 : gem install rdoc-data; rdoc-data --install
>= 1.9.2 : nothing to do! Yay!
Successfully installed rdoc-3.12.1
Gems updated: io-console, minitest, rake, rdoc
Installing ri documentation for io-console-0.4.1...
Installing ri documentation for minitest-4.5.0...
Installing ri documentation for rake-10.0.3...
Installing ri documentation for rdoc-3.12.1...
Installing RDoc documentation for io-console-0.4.1...
Installing RDoc documentation for minitest-4.5.0...
Installing RDoc documentation for rake-10.0.3...
Installing RDoc documentation for rdoc-3.12.1...
もう一度リトライ
gem install fluentd
Fetching: msgpack-0.4.7.gem (100%)
Building native extensions. This could take a while...
Fetching: iobuffer-1.1.2.gem (100%)
Building native extensions. This could take a while...
Fetching: cool.io-1.1.0.gem (100%)
Building native extensions. This could take a while...
Fetching: http_parser.rb-0.5.3.gem (100%)
Building native extensions. This could take a while...
Fetching: fluentd-0.10.31.gem (100%)
Successfully installed msgpack-0.4.7
Successfully installed iobuffer-1.1.2
Successfully installed cool.io-1.1.0
Successfully installed http_parser.rb-0.5.3
Successfully installed fluentd-0.10.31
5 gems installed
Installing ri documentation for msgpack-0.4.7...
Installing ri documentation for iobuffer-1.1.2...
Installing ri documentation for cool.io-1.1.0...
Installing ri documentation for http_parser.rb-0.5.3...
Installing ri documentation for fluentd-0.10.31...
Installing RDoc documentation for msgpack-0.4.7...
Installing RDoc documentation for iobuffer-1.1.2...
Installing RDoc documentation for cool.io-1.1.0...
Installing RDoc documentation for http_parser.rb-0.5.3...
Installing RDoc documentation for fluentd-0.10.31... #入った
※コマンドで最初fluentdが見つからなくて困ったが一旦virtualboxからexitで抜けて入りなおしたら入っていた。
多分PATHの設定がrbenvだと一旦コンソールから抜けないとダメっぽい。。
(というかinitの処理が必要だったみたい)
fluentdのセットアップ
まあ/etc/以下にあればいいだろうということで
fluentd --setup /etc/fluent
installed ./fluent/fluent.conf.
動作検証
これもマニュアルどおりにやってみる
fluentd -c /etc/fluent/fluent.conf -vv &
[1] 4075
[root@localhost fluent]# 2013-02-15 11:45:51 +0900: fluent/supervisor.rb:153:supervise: starting fluentd-0.10.31
2013-02-15 11:45:51 +0900: fluent/supervisor.rb:235:read_config: reading config file path="fluent.conf"
2013-02-15 11:45:51 +0900: fluent/plugin.rb:89:register_impl: registered buffer plugin 'file'
2013-02-15 11:45:51 +0900: fluent/plugin.rb:89:register_impl: registered buffer plugin 'memory'
2013-02-15 11:45:51 +0900: fluent/plugin.rb:89:register_impl: registered input plugin 'debug_agent'
2013-02-15 11:45:51 +0900: fluent/plugin.rb:89:register_impl: registered input plugin 'exec'
2013-02-15 11:45:51 +0900: fluent/plugin.rb:89:register_impl: registered input plugin 'forward'
2013-02-15 11:45:51 +0900: fluent/plugin.rb:89:register_impl: registered input plugin 'gc_stat'
2013-02-15 11:45:51 +0900: fluent/plugin.rb:89:register_impl: registered input plugin 'http'
2013-02-15 11:45:51 +0900: fluent/plugin.rb:89:register_impl: registered input plugin 'object_space'
2013-02-15 11:45:51 +0900: fluent/plugin.rb:89:register_impl: registered input plugin 'status'
2013-02-15 11:45:51 +0900: fluent/plugin.rb:89:register_impl: registered input plugin 'tcp'
2013-02-15 11:45:51 +0900: fluent/plugin.rb:89:register_impl: registered input plugin 'unix'
2013-02-15 11:45:51 +0900: fluent/plugin.rb:89:register_impl: registered input plugin 'syslog'
2013-02-15 11:45:51 +0900: fluent/plugin.rb:89:register_impl: registered input plugin 'tail'
2013-02-15 11:45:51 +0900: fluent/plugin.rb:89:register_impl: registered output plugin 'copy'
2013-02-15 11:45:51 +0900: fluent/plugin.rb:89:register_impl: registered output plugin 'exec'
2013-02-15 11:45:51 +0900: fluent/plugin.rb:89:register_impl: registered output plugin 'exec_filter'
2013-02-15 11:45:51 +0900: fluent/plugin.rb:89:register_impl: registered output plugin 'file'
2013-02-15 11:45:51 +0900: fluent/plugin.rb:89:register_impl: registered output plugin 'forward'
2013-02-15 11:45:51 +0900: fluent/plugin.rb:89:register_impl: registered output plugin 'null'
2013-02-15 11:45:51 +0900: fluent/plugin.rb:89:register_impl: registered output plugin 'roundrobin'
2013-02-15 11:45:51 +0900: fluent/plugin.rb:89:register_impl: registered output plugin 'stdout'
2013-02-15 11:45:51 +0900: fluent/plugin.rb:89:register_impl: registered output plugin 'tcp'
2013-02-15 11:45:51 +0900: fluent/plugin.rb:89:register_impl: registered output plugin 'unix'
2013-02-15 11:45:51 +0900: fluent/plugin.rb:89:register_impl: registered output plugin 'test'
2013-02-15 11:45:51 +0900: fluent/engine.rb:61:configure: using configuration file: <ROOT>
<source>
type forward
</source>
<source>
type http
port 8888
</source>
<source>
type debug_agent
port 24230
</source>
<match debug.**>
type stdout
</match>
</ROOT>
2013-02-15 11:45:51 +0900: fluent/engine.rb:70:block in configure: adding source type="forward"
2013-02-15 11:45:51 +0900: fluent/engine.rb:70:block in configure: adding source type="http"
2013-02-15 11:45:51 +0900: fluent/engine.rb:70:block in configure: adding source type="debug_agent"
2013-02-15 11:45:51 +0900: fluent/engine.rb:86:block in configure: adding match pattern="debug.**" type="stdout"
2013-02-15 11:45:51 +0900: plugin/in_forward.rb:63:listen: listening fluent socket on 0.0.0.0:24224
2013-02-15 11:45:51 +0900: plugin/in_http.rb:74:start: listening http on 0.0.0.0:8888
2013-02-15 11:45:51 +0900: plugin/in_debug_agent.rb:46:start: listening dRuby uri="druby://0.0.0.0:24230" object="Engine"
実際やってみる
echo '{"json":"message"}' | fluent-cat debug.test
2013-02-15 11:46:08 +0900: plugin/in_forward.rb:147:initialize: accepted fluent socket object_id=14395860
2013-02-15 11:46:08 +0900 debug.test: {"json":"message"}
2013-02-15 11:46:08 +0900: plugin/in_forward.rb:188:on_close: closed fluent socket object_id=14395860
動いてる
fluentdの起動と停止方法
起動
fluentd -c fluent.confのパス -vv &
停止
#デフォルトは24224っぽい
/usr/sbin/lsof | grep 24224
fluentd 6428 root 6u IPv4 44926 0t0 TCP *:24224 (LISTEN)
fluentd 6428 root 7u IPv4 44928 0t0 UDP *:24224
プロセスが生きてることを確認して
pkill fluentd
殺す!!
※一部のブログで下記の方法で停止をしていますが、たまにプロセスが重複起動してプロセスが殺しきれない場合があるので注意してください。
#まずはjobの確認
jobs -l
[1]+ 4075 Running fluentd -c fluent.conf -vv &
#そしてkill
kill 4075
# 2013-02-15 12:09:17 +0900: fluent/engine.rb:167:stop: shutting down fluentd
2013-02-15 12:09:18 +0900: fluent/supervisor.rb:170:supervise: process finished code=0
[1]+ Done fluentd -c fluent.conf -vv
apacheのアクセスログをjsonで出力する
/etc/fluent/fluent.conf
vim /etc/fluent/fluent.conf
#修正前
#<source>
# type tail
# format apache
# path /var/log/httpd/access_log
# tag apache.access
#</source>
#コメントアウトを外してpathの部分を変更する
#修正後
<source>
type tail
format apache
path /var/log/httpd/access_log
tag apache.access
</source>
# 修正前
#<match apache.access>
# type file
# path /var/log/fluent/access
#</match>
#修正後
<match apache.access>
type file
path /var/log/fluent/access
</match>
fluentdの再起動
再起動しないと読み込まなかったので
停止
pkill fluentd
再起動
fluentd -c /etc/fluent/fluent.conf -vv &
[1] 4171
[root@localhost fluent]# 2013-02-15 12:21:53 +0900: fluent/supervisor.rb:153:supervise: starting fluentd-0.10.31
2013-02-15 12:21:53 +0900: fluent/supervisor.rb:235:read_config: reading config file path="/etc/fluent/fluent.conf"
2013-02-15 12:21:53 +0900: fluent/plugin.rb:89:register_impl: registered buffer plugin 'file'
2013-02-15 12:21:53 +0900: fluent/plugin.rb:89:register_impl: registered buffer plugin 'memory'
2013-02-15 12:21:53 +0900: fluent/plugin.rb:89:register_impl: registered input plugin 'debug_agent'
2013-02-15 12:21:53 +0900: fluent/plugin.rb:89:register_impl: registered input plugin 'exec'
2013-02-15 12:21:53 +0900: fluent/plugin.rb:89:register_impl: registered input plugin 'forward'
2013-02-15 12:21:53 +0900: fluent/plugin.rb:89:register_impl: registered input plugin 'gc_stat'
2013-02-15 12:21:53 +0900: fluent/plugin.rb:89:register_impl: registered input plugin 'http'
2013-02-15 12:21:53 +0900: fluent/plugin.rb:89:register_impl: registered input plugin 'object_space'
2013-02-15 12:21:53 +0900: fluent/plugin.rb:89:register_impl: registered input plugin 'status'
2013-02-15 12:21:53 +0900: fluent/plugin.rb:89:register_impl: registered input plugin 'tcp'
2013-02-15 12:21:53 +0900: fluent/plugin.rb:89:register_impl: registered input plugin 'unix'
2013-02-15 12:21:53 +0900: fluent/plugin.rb:89:register_impl: registered input plugin 'syslog'
2013-02-15 12:21:53 +0900: fluent/plugin.rb:89:register_impl: registered input plugin 'tail'
2013-02-15 12:21:53 +0900: fluent/plugin.rb:89:register_impl: registered output plugin 'copy'
2013-02-15 12:21:53 +0900: fluent/plugin.rb:89:register_impl: registered output plugin 'exec'
2013-02-15 12:21:53 +0900: fluent/plugin.rb:89:register_impl: registered output plugin 'exec_filter'
2013-02-15 12:21:53 +0900: fluent/plugin.rb:89:register_impl: registered output plugin 'file'
2013-02-15 12:21:53 +0900: fluent/plugin.rb:89:register_impl: registered output plugin 'forward'
2013-02-15 12:21:53 +0900: fluent/plugin.rb:89:register_impl: registered output plugin 'null'
2013-02-15 12:21:53 +0900: fluent/plugin.rb:89:register_impl: registered output plugin 'roundrobin'
2013-02-15 12:21:53 +0900: fluent/plugin.rb:89:register_impl: registered output plugin 'stdout'
2013-02-15 12:21:53 +0900: fluent/plugin.rb:89:register_impl: registered output plugin 'tcp'
2013-02-15 12:21:53 +0900: fluent/plugin.rb:89:register_impl: registered output plugin 'unix'
2013-02-15 12:21:53 +0900: fluent/plugin.rb:89:register_impl: registered output plugin 'test'
2013-02-15 12:21:53 +0900: fluent/engine.rb:61:configure: using configuration file: <ROOT>
<source>
type forward
</source>
<source>
type http
port 8888
</source>
<source>
type tail
format apache
path /var/log/httpd/access_log
tag apache.access
</source>
<source>
type debug_agent
port 24230
</source>
<match apache.access>
type file
path /var/log/fluent/access
</match>
<match debug.**>
type stdout
</match>
</ROOT>
2013-02-15 12:21:53 +0900: fluent/engine.rb:70:block in configure: adding source type="forward"
2013-02-15 12:21:53 +0900: fluent/engine.rb:70:block in configure: adding source type="http"
2013-02-15 12:21:53 +0900: fluent/engine.rb:70:block in configure: adding source type="tail"
2013-02-15 12:21:53 +0900: plugin/in_tail.rb:49:configure: 'pos_file PATH' parameter is not set to a 'tail' source.
2013-02-15 12:21:53 +0900: plugin/in_tail.rb:50:configure: this parameter is highly recommended to save the position to resume tailing.
2013-02-15 12:21:53 +0900: fluent/engine.rb:70:block in configure: adding source type="debug_agent"
2013-02-15 12:21:53 +0900: fluent/engine.rb:86:block in configure: adding match pattern="apache.access" type="file"
2013-02-15 12:21:53 +0900: fluent/engine.rb:86:block in configure: adding match pattern="debug.**" type="stdout"
2013-02-15 12:21:53 +0900: plugin/in_forward.rb:63:listen: listening fluent socket on 0.0.0.0:24224
2013-02-15 12:21:53 +0900: plugin/in_http.rb:74:start: listening http on 0.0.0.0:8888
2013-02-15 12:21:53 +0900: plugin/in_tail.rb:289:initialize: following tail of /var/log/httpd/access_log
2013-02-15 12:21:53 +0900: plugin/in_debug_agent.rb:46:start: listening dRuby uri="druby://0.0.0.0:24230" object="Engine"
エラーとかがでていないことを確認する
出力内容の確認
場所は/var/log/fluent/httpd/access.日付.トークン
tail -f /var/log/fluent/httpd/access.20130215.b4d5bdcc431c4eeec
開発環境等にアクセスして監視してみる
例) http://localhostなどに複数回アクセスしてみる
出力されていればOK!!
後は起動スクリプトとかあったら便利だけど、心が折れたのでここまで!!
Discussion