🙆

fluentdを使ってTwitterデータを取得してみる

2023/03/04に公開

Fluentd

tech

fluentdを使ってTwitterのツイートデータを取得してみようと思います。
fluentdのインストール方法はこちらを参照してください。

twitter用プラグインのインストール

張り切ってインストールコマンドを実行したのですが、敢えなくエラーとなりました。。。

ubuntu@embulk:~$ sudo /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-twitter
Fetching: simple_oauth-0.3.1.gem (100%)
Successfully installed simple_oauth-0.3.1
Fetching: naught-1.1.0.gem (100%)
Successfully installed naught-1.1.0
Fetching: thread_safe-0.3.6.gem (100%)
Successfully installed thread_safe-0.3.6
Fetching: memoizable-0.4.2.gem (100%)
Successfully installed memoizable-0.4.2
Fetching: http-form_data-2.3.0.gem (100%)
Successfully installed http-form_data-2.3.0
Fetching: unf_ext-0.0.8.2.gem (100%)
Building native extensions.  This could take a while...
ERROR:  Error installing fluent-plugin-twitter:
	ERROR: Failed to build gem native extension.

    current directory: /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/unf_ext-0.0.8.2/ext/unf_ext
/opt/td-agent/embedded/bin/ruby -r ./siteconf20230225-40223-8kqdw6.rb extconf.rb
checking for -lstdc++... *** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.

Provided configuration options:
	--with-opt-dir
	--with-opt-include
	--without-opt-include=${opt-dir}/include
	--with-opt-lib
	--without-opt-lib=${opt-dir}/lib
	--with-make-prog
	--without-make-prog
	--srcdir=.
	--curdir
	--ruby=/opt/td-agent/embedded/bin/$(RUBY_BASE_NAME)
	--with-static-libstdc++
	--without-static-libstdc++
	--with-stdc++lib
	--without-stdc++lib
/opt/td-agent/embedded/lib/ruby/2.4.0/mkmf.rb:468:in `try_do': The compiler failed to generate an executable file. (RuntimeError)
You have to install development tools first.
	from /opt/td-agent/embedded/lib/ruby/2.4.0/mkmf.rb:553:in `try_link0'
	from /opt/td-agent/embedded/lib/ruby/2.4.0/mkmf.rb:568:in `try_link'
	from /opt/td-agent/embedded/lib/ruby/2.4.0/mkmf.rb:787:in `try_func'
	from /opt/td-agent/embedded/lib/ruby/2.4.0/mkmf.rb:1014:in `block in have_library'
	from /opt/td-agent/embedded/lib/ruby/2.4.0/mkmf.rb:957:in `block in checking_for'
	from /opt/td-agent/embedded/lib/ruby/2.4.0/mkmf.rb:351:in `block (2 levels) in postpone'
	from /opt/td-agent/embedded/lib/ruby/2.4.0/mkmf.rb:321:in `open'
	from /opt/td-agent/embedded/lib/ruby/2.4.0/mkmf.rb:351:in `block in postpone'
	from /opt/td-agent/embedded/lib/ruby/2.4.0/mkmf.rb:321:in `open'
	from /opt/td-agent/embedded/lib/ruby/2.4.0/mkmf.rb:347:in `postpone'
	from /opt/td-agent/embedded/lib/ruby/2.4.0/mkmf.rb:956:in `checking_for'
	from /opt/td-agent/embedded/lib/ruby/2.4.0/mkmf.rb:1009:in `have_library'
	from extconf.rb:6:in `<main>'

To see why this extension failed to compile, please check the mkmf.log which can be found here:

  /opt/td-agent/embedded/lib/ruby/gems/2.4.0/extensions/x86_64-linux/2.4.0/unf_ext-0.0.8.2/mkmf.log

extconf failed, exit code 1

Gem files will remain installed in /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/unf_ext-0.0.8.2 for inspection.
Results logged to /opt/td-agent/embedded/lib/ruby/gems/2.4.0/extensions/x86_64-linux/2.4.0/unf_ext-0.0.8.2/gem_make.out

対処方法

Railsでも同じエラーが出て解決された記事がありましたので参考にさせていただきました。
ありがとうございます。

$ sudo apt update -y
$ sudo apt install g++ make

再度プラグインのインストールにチャレンジ！

ubuntu@embulk:~$ sudo /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-twitter
Building native extensions.  This could take a while...
Successfully installed unf_ext-0.0.8.2
Fetching: unf-0.1.4.gem (100%)
Successfully installed unf-0.1.4
Fetching: domain_name-0.5.20190701.gem (100%)
Successfully installed domain_name-0.5.20190701
Fetching: http-cookie-1.0.5.gem (100%)
Successfully installed http-cookie-1.0.5
Fetching: http-3.3.0.gem (100%)
Successfully installed http-3.3.0
Fetching: equalizer-0.0.11.gem (100%)
Successfully installed equalizer-0.0.11
Fetching: buftok-0.2.0.gem (100%)
Successfully installed buftok-0.2.0
Fetching: twitter-6.2.0.gem (100%)
Successfully installed twitter-6.2.0
Fetching: fluent-plugin-twitter-0.6.1.gem (100%)
Successfully installed fluent-plugin-twitter-0.6.1
Parsing documentation for unf_ext-0.0.8.2
Installing ri documentation for unf_ext-0.0.8.2
Parsing documentation for unf-0.1.4
Installing ri documentation for unf-0.1.4
Parsing documentation for domain_name-0.5.20190701
Installing ri documentation for domain_name-0.5.20190701
Parsing documentation for http-cookie-1.0.5
Installing ri documentation for http-cookie-1.0.5
Parsing documentation for http-3.3.0
Installing ri documentation for http-3.3.0
Parsing documentation for equalizer-0.0.11
Installing ri documentation for equalizer-0.0.11
Parsing documentation for buftok-0.2.0
Installing ri documentation for buftok-0.2.0
Parsing documentation for twitter-6.2.0
Installing ri documentation for twitter-6.2.0
Parsing documentation for fluent-plugin-twitter-0.6.1
Installing ri documentation for fluent-plugin-twitter-0.6.1
Done installing documentation for unf_ext, unf, domain_name, http-cookie, http, equalizer, buftok, twitter, fluent-plugin-twitter after 14 seconds
9 gems installed
ubuntu@embulk:~$

無事にインストールできました。

td-agent.confの編集

以下をtd-agent.confに追記します。

<source>
  @type twitter
  consumer_key {consumer_key}
  consumer_secret {consumer_secret}
  access_token {access_token}
  access_token_secret {access_token_secret}
  tag input.twitter.sampling
  timeline sampling
  lang ja
  keyword 'Twitterで検索したキーワード'
  output_format nest
</source>

<match input.twitter.sampling>
  @type stdout
</match>

fluentdの起動

sudo service td-agent start

数秒ほど待ってから停止します。

sudo service td-agent stop

今回は検索キーワードを「マンダロリアン」にしていたので以下のとおりTweetを取得できました。
出力先はログは/var/log/td-agent/td-agent.logです。

今回はTwitter APIを使ってfluentdでTweetを取得してみました。
次回は取得したTweetデータをJSON形式で保存し、Snowflakeに取り込んでみたいと思います。

twitter用プラグインのインストール

対処方法

td-agent.confの編集

fluentdの起動

Discussion