Open7

TensorFlow 学習メモ

monrymonry

poetry とやらが良いらしい？

公式サイトの方法に従ってインストール

curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -

プロジェクトの初期化。

poetry new tutorial

% tree tutorial
.
├── README.rst
├── poetry.lock
├── pyproject.toml
├── tutorial
│   └── __init__.py
└── tests
    ├── __init__.py
    └── test_tutorial.py

どうやら プロジェクト (tutorial) > モジュール (tutorial) という構成になるっぽい。

monrymonry

Python はプロジェクト毎に仮想環境を作るのがお作法っぽい。

poetry new の時点で仮想環境作ってくれるっぽい？が、python3 -m venv .venv すると良いよって記事を見掛けたので従ってみる。

poetry config virtualenvs.in-project true
python3 -m venv .venv
poetry install

確認。

poetry env list
# .venv (Activated) ← こうなってれば OK っぽい

in-project の設定をする前に install しちゃった場合は venv 設定後に env を消すと良さそう？
（消さなくても良いのかな？）

monrymonry

TensorFlow 追加。

poetry add tensorflow

Using version ^2.7.0 for tensorflow

Updating dependencies
Resolving dependencies... (10.6s)
Resolving dependencies... (10.7s)
  SolverProblemError

  The current project's Python requirement (>=3.9,<4.0) is not compatible with some of the required packages Python requirement:
    - tensorflow-io-gcs-filesystem requires Python >=3.7, <3.11, so it will not be satisfied for Python >=3.11,<4.0
    - tensorflow-io-gcs-filesystem requires Python >=3.6, <3.10, so it will not be satisfied for Python >=3.10,<4.0
    - tensorflow-io-gcs-filesystem requires Python >=3.6, <3.10, so it will not be satisfied for Python >=3.10,<4.0
    - tensorflow-io-gcs-filesystem requires Python >=3.6, <3.10, so it will not be satisfied for Python >=3.10,<4.0

  Because no versions of tensorflow-io-gcs-filesystem match >0.21.0,<0.22.0 || >0.22.0,<0.23.0 || >0.23.0,<0.23.1 || >0.23.1
   and tensorflow-io-gcs-filesystem (0.21.0) requires Python >=3.6, <3.10, tensorflow-io-gcs-filesystem is forbidden.
  And because tensorflow-io-gcs-filesystem (0.22.0) requires Python >=3.6, <3.10, tensorflow-io-gcs-filesystem is forbidden.
  And because tensorflow-io-gcs-filesystem (0.23.0) requires Python >=3.6, <3.10
   and tensorflow-io-gcs-filesystem (0.23.1) requires Python >=3.7, <3.11, tensorflow-io-gcs-filesystem is forbidden.
  Because no versions of tensorflow match >2.7.0,<3.0.0
   and tensorflow (2.7.0) depends on tensorflow-io-gcs-filesystem (>=0.21.0), tensorflow (>=2.7.0,<3.0.0) requires tensorflow-io-gcs-filesystem (>=0.21.0).
  Thus, tensorflow is forbidden.
  So, because tutorial depends on tensorflow (^2.7.0), version solving failed.

  at ~/.poetry/lib/poetry/puzzle/solver.py:241 in _solve
      237│             packages = result.packages
      238│         except OverrideNeeded as e:
      239│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
      240│         except SolveFailure as e:
    → 241│             raise SolverProblemError(e)
      242│
      243│         results = dict(
      244│             depth_first_search(
      245│                 PackageNode(self._package, packages), aggregate_package_nodes

  • Check your dependencies Python requirement: The Python requirement can be specified via the `python` or `markers` properties

    For tensorflow-io-gcs-filesystem, a possible solution would be to set the `python` property to ">=3.9,<3.11"
    For tensorflow-io-gcs-filesystem, a possible solution would be to set the `python` property to ">=3.9,<3.10"
    For tensorflow-io-gcs-filesystem, a possible solution would be to set the `python` property to ">=3.9,<3.10"
    For tensorflow-io-gcs-filesystem, a possible solution would be to set the `python` property to ">=3.9,<3.10"

    https://python-poetry.org/docs/dependency-specification/#python-restricted-dependencies,
    https://python-poetry.org/docs/dependency-specification/#using-environment-markers

ガッツリ怒られるｗ

どうやら Python のバージョン指定方法がマズいっぽい。

pyproject.toml の Python バージョンの指定を ^3.9 から >=3.9,<3.10 に変更してみる。

 [tool.poetry.dependencies]
-python = "^3.9"
+python = ">=3.9,<3.10"

再実行。

poetry add tensorflow

Using version ^2.7.0 for tensorflow

Updating dependencies
Resolving dependencies... (0.9s)

Writing lock file

Package operations: 40 installs, 0 updates, 0 removals

  • Installing certifi (2021.10.8)
  • Installing charset-normalizer (2.0.9)
  • Installing idna (3.3)
  • Installing pyasn1 (0.4.8)
  • Installing urllib3 (1.26.7)
  • Installing cachetools (4.2.4)
  • Installing oauthlib (3.1.1)
  • Installing pyasn1-modules (0.2.8)
  • Installing requests (2.26.0)
  • Installing rsa (4.8)
  • Installing six (1.16.0)
  • Installing zipp (3.6.0)
  • Installing google-auth (2.3.3)
  • Installing importlib-metadata (4.10.0)
  • Installing requests-oauthlib (1.3.0)
  • Installing absl-py (1.0.0)
  • Installing google-auth-oauthlib (0.4.6)
  • Installing grpcio (1.43.0)
  • Installing markdown (3.3.6)
  • Installing numpy (1.21.5)
  • Installing protobuf (3.19.1)
  • Installing tensorboard-data-server (0.6.1)
  • Installing tensorboard-plugin-wit (1.8.0)
  • Installing werkzeug (2.0.2)
  • Installing astunparse (1.6.3)
  • Installing flatbuffers (2.0)
  • Installing gast (0.4.0)
  • Installing google-pasta (0.2.0)
  • Installing h5py (3.6.0)
  • Installing keras (2.7.0)
  • Installing keras-preprocessing (1.1.2)
  • Installing libclang (12.0.0)
  • Installing opt-einsum (3.3.0)
  • Installing tensorboard (2.7.0)
  • Installing tensorflow-estimator (2.7.0)
  • Installing tensorflow-io-gcs-filesystem (0.23.1)
  • Installing termcolor (1.1.0)
  • Installing typing-extensions (4.0.1)
  • Installing wrapt (1.13.3)
  • Installing tensorflow (2.7.0)

成功 🎉

monrymonry

PyCharm で開いてみる。


ちゃんと認識してる

とりあえず読み込めているようす。


No Interpreter

だけど、インタプリタが設定できていないっぽい。


Add Interpreter から Poetry のインタプリタを設定

Poetry のを指定するべきかどうかは分からんけど、とりあえず設定した。

monrymonry

続いて、Jupyter Notebook によるコーディングを試したいので jupyter をインストールする。

poetry add jupyter

Using version ^1.0.0 for jupyter

Updating dependencies
Resolving dependencies... (3.2s)

Writing lock file

Package operations: 52 installs, 0 updates, 0 removals

  • Installing pyrsistent (0.18.0)
  • Installing traitlets (5.1.1)
  • Installing entrypoints (0.3)
  • Installing ipython-genutils (0.2.0)
  • Installing jsonschema (4.3.2)
  • Installing jupyter-core (4.9.1)
  • Installing nest-asyncio (1.5.4)
  • Installing parso (0.8.3)
  • Installing ptyprocess (0.7.0)
  • Installing pycparser (2.21)
  • Installing python-dateutil (2.8.2)
  • Installing pyzmq (22.3.0)
  • Installing tornado (6.1)
  • Installing appnope (0.1.2)
  • Installing backcall (0.2.0)
  • Installing cffi (1.15.0)
  • Installing decorator (5.1.0)
  • Installing jedi (0.18.1)
  • Installing jupyter-client (7.1.0)
  • Installing markupsafe (2.0.1)
  • Installing matplotlib-inline (0.1.3)
  • Installing nbformat (5.1.3)
  • Installing pexpect (4.8.0)
  • Installing pickleshare (0.7.5)
  • Installing prompt-toolkit (3.0.24)
  • Installing pygments (2.10.0)
  • Installing webencodings (0.5.1)
  • Installing argon2-cffi-bindings (21.2.0)
  • Installing bleach (4.1.0)
  • Installing debugpy (1.5.1)
  • Installing defusedxml (0.7.1)
  • Installing ipython (7.30.1)
  • Installing jinja2 (3.0.3)
  • Installing jupyterlab-pygments (0.1.2)
  • Installing mistune (0.8.4)
  • Installing nbclient (0.5.9)
  • Installing pandocfilters (1.5.0)
  • Installing testpath (0.5.0)
  • Installing argon2-cffi (21.3.0)
  • Installing ipykernel (6.6.0)
  • Installing nbconvert (6.3.0)
  • Installing prometheus-client (0.12.0)
  • Installing send2trash (1.8.0)
  • Installing terminado (0.12.1)
  • Installing notebook (6.4.6)
  • Installing jupyterlab-widgets (1.0.2)
  • Installing qtpy (2.0.0)
  • Installing widgetsnbextension (3.5.2)
  • Installing ipywidgets (7.6.5)
  • Installing jupyter-console (6.4.0)
  • Installing qtconsole (5.2.2)
  • Installing jupyter (1.0.0)
monrymonry

TensorFlow Tutorials の Beginner のヤツを写経してみる。


補完候補に tf.keras.datasets が出てこない…。

型ヒントのファイル？らしき .pyi が無いから補完されないのかな…？

find .venv -name "*tensorflow*"

.venv/lib/python3.9/site-packages/opt_einsum/backends/__pycache__/tensorflow.cpython-39.pyc
.venv/lib/python3.9/site-packages/opt_einsum/backends/tensorflow.py
.venv/lib/python3.9/site-packages/tensorflow_estimator
.venv/lib/python3.9/site-packages/tensorflow_io_gcs_filesystem
.venv/lib/python3.9/site-packages/tensorflow_io_gcs_filesystem/core/python/ops/libtensorflow_io_gcs_filesystem.so
.venv/lib/python3.9/site-packages/tensorflow_estimator-2.7.0.dist-info
.venv/lib/python3.9/site-packages/tensorflow
.venv/lib/python3.9/site-packages/tensorflow/core/protobuf/tensorflow_server_pb2.py
.venv/lib/python3.9/site-packages/tensorflow/core/protobuf/__pycache__/tensorflow_server_pb2.cpython-39.pyc
.venv/lib/python3.9/site-packages/tensorflow/python/__pycache__/pywrap_tensorflow.cpython-39.pyc
.venv/lib/python3.9/site-packages/tensorflow/python/__pycache__/pywrap_tensorflow_internal.cpython-39.pyc
.venv/lib/python3.9/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so
.venv/lib/python3.9/site-packages/tensorflow/python/pywrap_tensorflow_internal.py
.venv/lib/python3.9/site-packages/tensorflow/python/pywrap_tensorflow.py
.venv/lib/python3.9/site-packages/tensorflow/python/debug/lib/__pycache__/grpc_tensorflow_server.cpython-39.pyc
.venv/lib/python3.9/site-packages/tensorflow/python/debug/lib/grpc_tensorflow_server.py
.venv/lib/python3.9/site-packages/tensorflow/lite/python/optimize/_pywrap_tensorflow_lite_calibration_wrapper.so
.venv/lib/python3.9/site-packages/tensorflow/lite/python/metrics_wrapper/_pywrap_tensorflow_lite_metrics_wrapper.so
.venv/lib/python3.9/site-packages/tensorflow/lite/python/interpreter_wrapper/_pywrap_tensorflow_interpreter_wrapper.so
.venv/lib/python3.9/site-packages/tensorflow/include/tensorflow
.venv/lib/python3.9/site-packages/tensorflow/include/tensorflow/core/protobuf/tensorflow_server.proto
.venv/lib/python3.9/site-packages/tensorflow/include/tensorflow/core/protobuf/tensorflow_server.pb.h
.venv/lib/python3.9/site-packages/tensorflow/include/tensorflow/compiler/mlir/tensorflow
.venv/lib/python3.9/site-packages/tensorflow/xla_aot_runtime_src/tensorflow
.venv/lib/python3.9/site-packages/tensorflow/libtensorflow_framework.2.dylib
.venv/lib/python3.9/site-packages/tensorflow/libtensorflow_framework.2.7.0.dylib
.venv/lib/python3.9/site-packages/tensorflow/compiler/mlir/tensorflow
.venv/lib/python3.9/site-packages/tensorflow/libtensorflow_framework.dylib
.venv/lib/python3.9/site-packages/tensorboard/compat/tensorflow_stub
.venv/lib/python3.9/site-packages/tensorboard/compat/tensorflow_stub/__pycache__/pywrap_tensorflow.cpython-39.pyc
.venv/lib/python3.9/site-packages/tensorboard/compat/tensorflow_stub/pywrap_tensorflow.py
.venv/lib/python3.9/site-packages/tensorflow_io_gcs_filesystem-0.23.1.dist-info
.venv/lib/python3.9/site-packages/tensorflow-2.7.0.dist-info
.venv/lib/python3.9/site-packages/tensorboard_plugin_wit/_vendor/tensorflow_serving

Google Colaboratory だと補完されてたのは何だったんだろ…？

monrymonry

sklearn と pandas も追加

poetry add sklearn
poetry add pandas
# 出力は省略

sklearn は train_test_split を使うために入れる。
from sklearn.model_selection import train_test_split などとしてインポートするのが常套句？
（TensorFlow 側とかにあっても良さそうなもんだけど、そういう世界の話でもないのかな？）

pandas は Python で構造化されたファイルを読み込むときの便利ライブラリらしい。
import pandas as pd とかでインポートするのが一般的だそうな。
pd.read_csv() の戻り値は DataFrame 型なので df という変数で受けるコトが多いっぽい。