👖

Pants build system を使ってアプリケーションを作る

2023/10/30に公開

Python

pants

tech

Pants build system を使って、アプリケーションを作ります。

使用するバージョンは以下のとおりです。

Python 3.11.2
Pants 2.17.0

Check python version.

python -V

Output:

Python 3.11.2

Create a new directory

mkdir pants-build-study
cd pants-build-study

Write .gitignore.

cat <<EOF > .gitignore
venv
EOF

Install `pants`

curl --proto '=https' --tlsv1.2 -fsSL https://static.pantsbuild.org/setup/get-pants.sh | bash

1. First application

まずは簡単なコンソールアプリケーションを作ります。

Create `console-app` using poetry

mkdir -p src/python
cd src/python
poetry new console-app
cd console-app
python -m venv venv
. venv/bin/activate
pip install poetry
deactivate
. venv/bin/activate
poetry add pendulum
poetry install
deactivate

Write source codes

Write source code: src/python/console-app/console_app/main.py.

cat <<EOF > console_app/main.py
from console_app.util.util import now

print('console-app')
x = now()
print(x)
EOF

絶対パスでインポートできることを確認するためにもう 1 つコードを作成します。

Write source code: src/python/console-app/console_app/util/util.py.

mkdir -p console_app/util

cat <<EOF > console_app/util/util.py
import pendulum

def now():
    return pendulum.now('Europe/Paris')
EOF

cd ../../../

Create `pants.toml`

cat <<EOF > pants.toml
[GLOBAL]
pants_version = "2.17.0"

# https://www.pantsbuild.org/docs/enabling-backends
backend_packages = [
  "pants.backend.python",
]

[python]
interpreter_constraints = [">=3.11,<3.12"]

# https://www.pantsbuild.org/docs/source-roots
[source]
root_patterns = [
    "/src/python/*",
]
EOF

Create `BUILD` files using `pants tailor` command

pants tailor ::

Output:

Created src/python/console-app/BUILD:
  - Add poetry_requirements target poetry
Created src/python/console-app/console_app/BUILD:
  - Add python_sources target console_app
Created src/python/console-app/console_app/util/BUILD:
  - Add python_sources target util

Run application via source file

pants run src/python/console-app/console_app/main.py

Output:

console-app
2023-10-29T13:50:10.680589+01:00

Run application via binary file

Add below pex_binary target to src/python/console-app/BUILD.

pex_binary(
    name="console-app",
    entry_point="console_app/main.py",
    dependencies=[
        ":src",
    ],
)

cat <<EOF > src/python/console-app/BUILD
poetry_requirements(
    name="poetry",
)

python_sources(
    name="src",
    dependencies=[
        "src/python/console-app/console_app/**/*.py",
    ]
)

pex_binary(
    name="console-app",
    entry_point="console_app/main.py",
    dependencies=[
        ":src",
    ],
)
EOF

Run application via binary file.

pants run src/python/console-app:console-app

:console-app in the above command is related to the name of pex_binary in src/python/console-app/BUILD file.

ここまでで python ファイルを指定した場合とバイナリファイルで実行する場合の 2 つの方法でアプリケーションを実行することができました。

2. gRPC server

続いて、gRPC のサーバーアプリケーションを作ります。前回と異なるのは proto ファイルからのソースコードの生成です。

Create `grpc-server` using poetry.

cd src/python
poetry new grpc-server
cd grpc-server
python -m venv venv
. venv/bin/activate
pip install poetry
deactivate
. venv/bin/activate
poetry add grpcio
poetry add protobuf
poetry install
deactivate
cd ../../../

Create proto file.

mkdir -p src/protos/helloworld/v1

cat <<EOF > src/protos/helloworld/v1/helloworld.proto
// Copyright 2015 gRPC authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
//     http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

syntax = "proto3";

package helloworld.v1;

option java_multiple_files = true;
option java_outer_classname = "HelloWorldProto";
option java_package = "io.grpc.examples.helloworld";
option objc_class_prefix = "HLW";

// The greeting service definition.
service Greeter {
  // Sends a greeting
  rpc SayHello(HelloRequest) returns (HelloReply) {}
}

// The request message containing the user's name.
message HelloRequest {
  string name = 1;
}

// The response message containing the greetings
message HelloReply {
  string message = 1;
}
EOF

Update configurations.

Update pants.toml for generating source files from proto files.

Add "pants.backend.codegen.protobuf.python", to backend_packages in GLOBAL section.
Add "/src/protos", to root_patterns in source section.

cat <<EOF > pants.toml
[GLOBAL]
pants_version = "2.17.0"

backend_packages = [
    "pants.backend.python",
    "pants.backend.codegen.protobuf.python",
]

[python]
interpreter_constraints = [">=3.11,<3.12"]

[source]
root_patterns = [
    "/src/python/*",
    "/src/protos",
]
EOF

Create BUILD files using pants tailor command.

pants tailor ::

Output:

Created src/protos/helloworld/v1/BUILD:
  - Add protobuf_sources target v1
Created src/python/grpc-server/BUILD:
  - Add poetry_requirements target poetry

Update BUILD file to generate gRPC source files.

cat <<EOF > src/protos/helloworld/v1/BUILD
protobuf_sources(
    grpc=True,
)
EOF

Write initial source codes

Check whether app can import genarated source files.

cat <<EOF > src/python/grpc-server/grpc_server/main.py
from helloworld.v1.helloworld_pb2_grpc import GreeterServicer

print(GreeterServicer())
EOF

Create BUILD files using pants tailor command.

pants tailor ::

Test application.

pants run src/python/grpc-server/grpc_server/main.py

Output:

<helloworld.v1.helloworld_pb2_grpc.GreeterServicer object at 0x7fd0a35057d0>

Update source codes

cat <<EOF > src/python/grpc-server/grpc_server/main.py
from concurrent import futures

import grpc
from helloworld.v1 import helloworld_pb2, helloworld_pb2_grpc


class GreeterServicer(helloworld_pb2_grpc.GreeterServicer):
    def SayHello(
        self, request: helloworld_pb2.HelloRequest, context: grpc.ServicerContext
    ) -> helloworld_pb2.HelloReply:
        return helloworld_pb2.HelloReply(message=f"Hello, {request.name}")


server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
helloworld_pb2_grpc.add_GreeterServicer_to_server(GreeterServicer(), server)
server.add_insecure_port("[::]:50052")
server.start()
print("gRPC server listening at :50052")
server.wait_for_termination()
EOF

Run gRPC server

pants run src/python/grpc-server/grpc_server/main.py

Output:

gRPC server listening at :50052

Exec Ctrl+C command to stop the server application.

Create Docker image

Update BUILD file: src/python/grpc-server/BUILD.

cat <<EOF > src/python/grpc-server/BUILD
poetry_requirements(
    name="poetry",
)

pex_binary(
    name="grpc-server",
    entry_point="grpc_server/main.py",
)
EOF

rm -rf dist

pants package ::

Output:

00:35:43.16 [INFO] Wrote dist/src.python.console-app/console-app.pex
00:35:43.16 [INFO] Wrote dist/src.python.grpc-server/grpc-server.pex

Check whether a pex file is created.

ls dist/src.python.grpc-server/grpc-server.pex

Write Docker file.

cat <<EOF > src/python/grpc-server/Dockerfile
FROM python:3.11.2-slim-buster

WORKDIR /opt/app

COPY src.python.grpc-server/grpc-server.pex /opt/app/grpc_server.pex

ENTRYPOINT ["/bin/bash", "-c", "/opt/app/grpc_server.pex"]
EOF

Add "pants.backend.docker", to backend_packages in GLOBAL section in pants.toml.

cat <<EOF > pants.toml
[GLOBAL]
pants_version = "2.17.0"

backend_packages = [
    "pants.backend.python",
    "pants.backend.codegen.protobuf.python",
    "pants.backend.docker",
]

[python]
interpreter_constraints = [">=3.11,<3.12"]

[source]
root_patterns = [
    "/src/python/*",
    "/src/protos",
]
EOF

Add below docker_image target to src/python/grpc-server/BUILD.

docker_image(
    name="docker",
    repository="grpc-server",
)

cat <<EOF > src/python/grpc-server/BUILD
poetry_requirements(
    name="poetry",
)

python_sources(
    name="src",
    dependencies=[
        "src/python/grpc-server/grpc_server/**/*.py",
    ]
)

pex_binary(
    name="grpc-server",
    entry_point="grpc_server/main.py",
    dependencies=[
        ":src",
    ],
)

docker_image(
    name="docker",
    repository="grpc-server",
)
EOF

Build a docker image.

pants package ::

Check whether an image is created.

docker images

Output:

REPOSITORY     TAG       IMAGE ID       CREATED         SIZE
grpc-server    latest    5175e22508ed   4 minutes ago   127MB

3. Third application

Create another application that uses pendulum.

Check dependencies of `console-app`

pants peek src/python/console-app/console_app/util/util.py

Output:

[
  {
    "address": "src/python/console-app/console_app/util/util.py",
    "target_type": "python_source",
    "dependencies": [
      "src/python/console-app:poetry#pendulum"
    ],

You can see that console-app depends on src/python/console-app:poetry#pendulum.

Create `console2-app` using poetry

cd src/python
poetry new console2-app
cd console2-app
python -m venv venv
. venv/bin/activate
pip install poetry
deactivate
. venv/bin/activate
poetry add pendulum
poetry install
deactivate
cd ../../../

Write source codes

Write source code: src/python/console2-app/console2_app/main.py.

cat <<EOF > src/python/console2-app/console2_app/main.py
from console2_app.util.util import now

print('console2-app')
x = now()
print(x)
EOF

Write source code: src/python/console2-app/console2_app/util/util.py.

mkdir -p src/python/console2-app/console2_app/util

cat <<EOF > src/python/console2-app/console2_app/util/util.py
import pendulum

def now():
    return pendulum.now('Europe/Paris')
EOF

pants tailor ::

Run application

pants run src/python/console-app/console_app/main.py

Output:

Traceback (most recent call last):
  File "/tmp/pants-sandbox-n8l3my/./.cache/pex_root/venvs/4cc0b1a06847eddb3e2b2fca231d89d1240f8d9d/dd50d8563c27f406491b760155d53072745cd67f/pex", line 274, in <module>
    runpy.run_module(module_name, run_name="__main__", alter_sys=True)
  File "<frozen runpy>", line 226, in run_module
  File "<frozen runpy>", line 98, in _run_module_code
  File "<frozen runpy>", line 88, in _run_code
  File "/tmp/pants-sandbox-n8l3my/src/python/console-app/console_app/main.py", line 1, in <module>
    from console_app.util.util import now
  File "/tmp/pants-sandbox-n8l3my/src/python/console-app/console_app/util/util.py", line 1, in <module>
    import pendulum
ModuleNotFoundError: No module named 'pendulum'

Why did you get the above error?

Check dependencies of `console-app` again

pants peek src/python/console-app/console_app/util/util.py

You can see that dependencies are missing.

Output:

[
  {
    "address": "src/python/console-app/console_app/util/util.py",
    "target_type": "python_source",
    "dependencies": [],

To fix this issue, update src/python/console-app/console_app/util/BUILD file.

cat <<EOF > src/python/console-app/console_app/util/BUILD
python_sources(
    dependencies=[
        "src/python/console-app:poetry#pendulum",
    ]
)
EOF

Also update src/python/console2-app/console2_app/util/BUILD file.

cat <<EOF > src/python/console2-app/console2_app/util/BUILD
python_sources(
    dependencies=[
        "src/python/console2-app:poetry#pendulum",
    ]
)
EOF

Run application again

pants run src/python/console-app/console_app/main.py

Output:

console-app
2023-10-30T16:40:33.634042+01:00

Succeeded!

4. Fourth application

Use different versions of pendulum.

Create `old-app` using poetry.

cd src/python
poetry new old-app
cd old-app
python -m venv venv
. venv/bin/activate
pip install poetry
deactivate
. venv/bin/activate
poetry add pendulum=2.0.4
poetry install
deactivate
cd ../../../

Write source codes

Write source code: src/python/old-app/old_app/main.py.

cat <<EOF > src/python/old-app/old_app/main.py
from old_app.util.util import now

print('old-app')
x = now()
print(x)
EOF

Write source code: src/python/old-app/old_app/util/util.py.

mkdir -p src/python/old-app/old_app/util

cat <<EOF > src/python/old-app/old_app/util/util.py
import pendulum

def now():
    return pendulum.now('Europe/Paris')
EOF

pants tailor ::

Update src/python/old-app/old_app/util/BUILD file.

cat <<EOF > src/python/old-app/old_app/util/BUILD
python_sources(
    dependencies=[
        "src/python/old-app:poetry#pendulum",
    ]
)
EOF

pants run src/python/old-app/old_app/main.py

cat <<EOF > src/python/old-app/BUILD
poetry_requirements(
    name="poetry",
)

pex_binary(
    name="old-app",
    entry_point="old_app/main.py",
)
EOF

rm -rf dist
pants package ::

cd dist/src.python.old-app
unzip old-app.pex
ls .deps

Output:

pendulum-2.0.4-cp311-cp311-manylinux_2_35_x86_64.whl
python_dateutil-2.8.2-py2.py3-none-any.whl
pytzdata-2020.1-py2.py3-none-any.whl
six-1.16.0-py2.py3-none-any.whl

You can see that old-app refers 2.0.4 of pendulum.

cd ../../

5. gRPC client

Create `grpc-client` using poetry.

cd src/python
poetry new grpc-client
cd grpc-client
python -m venv venv
. venv/bin/activate
pip install poetry
deactivate
. venv/bin/activate
poetry add grpc-stubs
poetry install
deactivate
cd ../../../

Write source codes

Write source code: src/python/grpc-client/grpc_client/main.py.

cat <<EOF > src/python/grpc-client/grpc_client/main.py
import grpc
from helloworld.v1 import helloworld_pb2, helloworld_pb2_grpc


with grpc.insecure_channel("localhost:50052") as channel:
    stub = helloworld_pb2_grpc.GreeterStub(channel)
    response = stub.SayHello(helloworld_pb2.HelloRequest(name="John"))
print("Greeter client received: " + response.message)
EOF

pants tailor ::

cat <<EOF > src/python/grpc-client/BUILD
poetry_requirements(
    name="poetry",
)

pex_binary(
    name="grpc-client",
    entry_point="grpc_client/main.py",
)
EOF

rm -rf ./dist
pants package ::

Run gRPC server.

./dist/src.python.grpc-server/grpc-server.pex

Run gRPC client

Run gRPC client on another terminal.

./dist/src.python.grpc-client/grpc-client.pex

Output:

Greeter client received: Hello, John

6. Use Lockfiles

Add the below line to python section in pants.toml file.

enable_resolves = true

cat <<EOF > pants.toml
[GLOBAL]
pants_version = "2.17.0"

backend_packages = [
    "pants.backend.python",
    "pants.backend.codegen.protobuf.python",
    "pants.backend.docker",
]

[python]
interpreter_constraints = [">=3.11,<3.12"]
enable_resolves = true

[source]
root_patterns = [
    "/src/python/*",
    "/src/protos",
]
EOF

Generate lockfiles

pants generate-lockfiles ::

Output:

-resolution/#dealing-with-dependency-conflicts
 
 The conflict is caused by:
     The user requested pendulum<3.0.0 and >=2.1.2
     The user requested pendulum==2.0.4
 
 To fix this you could try to:
 1. loosen the range of package versions you've specified
 2. remove package versions to allow pip attempt to solve the dependency conflict

Use lockfile to specify version for each requirement set.

Generate lockfiles for old_app and others

cat <<EOF > pants.toml
[GLOBAL]
pants_version = "2.17.0"

backend_packages = [
    "pants.backend.python",
    "pants.backend.codegen.protobuf.python",
    "pants.backend.docker",
]

[python]
interpreter_constraints = [">=3.11,<3.12"]
enable_resolves = true
default_resolve = "default"

[python.resolves]
default = "3rdparty/python/default.lock"
old_app = "3rdparty/python/old_app.lock"

[source]
root_patterns = [
    "/src/python/*",
    "/src/protos",
]
EOF

Add resolve="old_app", to poetry_requirements and pex_binary target in src/python/old-app/BUILD file.

cat <<EOF > src/python/old-app/BUILD
poetry_requirements(
    name="poetry",
    resolve="old_app",
)

pex_binary(
    name="old-app",
    resolve="old_app",
    entry_point="old_app/main.py",
)
EOF

Add resolve="old_app", to python_sources target in all BUILD file.

cat <<EOF > src/python/old-app/old_app/BUILD
python_sources(
    resolve="old_app",
)
EOF

cat <<EOF > src/python/old-app/old_app/util/BUILD
python_sources(
    resolve="old_app",
)
EOF

pants generate-lockfiles ::

Output:

23:19:04.55 [INFO] Completed: Generate lockfile for default
23:19:17.80 [INFO] Completed: Generate lockfile for old_app
23:19:17.81 [INFO] Wrote lockfile for the resolve `default` to 3rdparty/python/default.lock
23:19:17.81 [INFO] Wrote lockfile for the resolve `old_app` to 3rdparty/python/old_app.lock

3rdparty/pyhon/default.lock and 3rdparty/pyhon/old_app.lock were created.

The below is the header of 3rdparty/pyhon/default.lock file.

// This lockfile was autogenerated by Pants. To regenerate, run:
//
//    pants generate-lockfiles --resolve=default
//
// --- BEGIN PANTS LOCKFILE METADATA: DO NOT EDIT OR REMOVE ---
// {
//   "version": 3,
//   "valid_for_interpreter_constraints": [
//     "CPython<3.12,>=3.11"
//   ],
//   "generated_with_requirements": [
//     "grpc-stubs<2.0.0,>=1.53.0.3",
//     "grpcio<2.0.0,>=1.59.0",
//     "pendulum<3.0.0,>=2.1.2",
//     "protobuf<5.0.0,>=4.24.4"
//   ],
//   "manylinux": "manylinux2014",
//   "requirement_constraints": [],
//   "only_binary": [],
//   "no_binary": []
// }
// --- END PANTS LOCKFILE METADATA ---

The below is the header of 3rdparty/pyhon/old_app.lock file.

// This lockfile was autogenerated by Pants. To regenerate, run:
//
//    pants generate-lockfiles --resolve=old_app
//
// --- BEGIN PANTS LOCKFILE METADATA: DO NOT EDIT OR REMOVE ---
// {
//   "version": 3,
//   "valid_for_interpreter_constraints": [
//     "CPython<3.12,>=3.11"
//   ],
//   "generated_with_requirements": [
//     "pendulum==2.0.4"
//   ],
//   "manylinux": "manylinux2014",
//   "requirement_constraints": [],
//   "only_binary": [],
//   "no_binary": []
// }
// --- END PANTS LOCKFILE METADATA ---

pants package ::

You can see that the build is successful.

7. Generating Source code from proto files for each resolve

Add `grpc_client` resolve

Downgrade the grpcio version to use different resolve.

cd src/python/grpc-client
. venv/bin/activate
poetry add grpcio=1.58.0
poetry add protobuf
poetry install
deactivate
cd ../../..

Update src/python/grpc-client/BUILD file

cat <<EOF > src/python/grpc-client/BUILD
poetry_requirements(
    name="poetry",
    resolve="grpc_client",
)

pex_binary(
    name="grpc-client",
    resolve="grpc_client",
    entry_point="grpc_client/main.py",
)
EOF

Add another `protobuf_sources` to BUILD file for proto files.

Add protobuf_sources to BUILD file for grpc_client resolve.

cat <<EOF > src/protos/helloworld/v1/BUILD
protobuf_sources(
    name="default",
    grpc=True,
)

protobuf_sources(
    name="grpc_client",
    grpc=True,
    python_resolve="grpc_client",
)
EOF

Update BUILD files for grpc-client application

Update src/python/grpc-client/grpc_client/BUILD file.

cat <<EOF > src/python/grpc-client/grpc_client/BUILD
python_sources(
    resolve="grpc_client",
)
EOF

Update pants.toml for grpc-client application

Add the below line to pants.toml

grpc_client = "3rdparty/python/grpc_client.lock"

cat <<EOF > pants.toml
[GLOBAL]
pants_version = "2.17.0"

backend_packages = [
    "pants.backend.python",
    "pants.backend.codegen.protobuf.python",
    "pants.backend.docker",
]

[python]
interpreter_constraints = [">=3.11,<3.12"]
enable_resolves = true
default_resolve = "default"

[python.resolves]
default = "3rdparty/python/default.lock"
old_app = "3rdparty/python/old_app.lock"
grpc_client = "3rdparty/python/grpc_client.lock"

[source]
root_patterns = [
    "/src/python/*",
    "/src/protos",
]
EOF

Update lockfiles.

pants generate-lockfiles ::

Check if grpc-client runs properly.

pants run src/python/grpc-client/grpc_client/main.py

pants run src/python/grpc-client:grpc-client

8. Linters and formatters

Add bandit

Add "pants.backend.python.lint.bandit", to backend_packages in GLOBAL seciton.

cat <<EOF >pants.toml
[GLOBAL]
pants_version = "2.17.0"

backend_packages = [
    "pants.backend.python",
    "pants.backend.python.lint.bandit",
    "pants.backend.codegen.protobuf.python",
    "pants.backend.docker",
]

[python]
interpreter_constraints = [">=3.11,<3.12"]
enable_resolves = true
default_resolve = "default"

[python.resolves]
default = "3rdparty/python/default.lock"
mypy = "3rdparty/python/mypy.lock"
old_app = "3rdparty/python/old_app.lock"
grpc_client = "3rdparty/python/grpc_client.lock"

[source]
root_patterns = [
    "/src/python/*",
    "/src/protos",
]
EOF

Apply linter.

pants lint ::

Add other linters

mkdir build-support

Add config file for pylint.

cat <<EOF >build-support/pylint.config
[MASTER]
disable=
    C0114, # missing-module-docstring
    C0115, # missing-class-docstring
    C0116, # missing-function-docstring
    E1101, # no-member
    R0903, # too-few-public-methods
EOF

Add config file for isort and black.

cat <<EOF >build-support/pyproject.toml
[tool.isort]
profile = "black"
line_length = 100

[tool.black]
line-length = 100
EOF

Add below lines to backend_packages in GLOBAL seciton in pants.toml file.

"pants.backend.python.lint.bandit",
"pants.backend.python.lint.black",
"pants.backend.python.lint.docformatter",
"pants.backend.docker.lint.hadolint",
"pants.backend.python.lint.isort",
"pants.backend.python.lint.pylint",
"pants.backend.python.lint.pyupgrade",
"pants.backend.experimental.python.lint.ruff",

cat <<EOF >pants.toml
[GLOBAL]
pants_version = "2.17.0"

backend_packages = [
    "pants.backend.python",
    "pants.backend.python.lint.bandit",
    "pants.backend.python.lint.black",
    "pants.backend.python.lint.docformatter",
    "pants.backend.docker.lint.hadolint",
    "pants.backend.python.lint.isort",
    "pants.backend.python.lint.pylint",
    "pants.backend.python.lint.pyupgrade",
    "pants.backend.experimental.python.lint.ruff",
    "pants.backend.codegen.protobuf.python",
    "pants.backend.docker",
]

[python]
interpreter_constraints = [">=3.11,<3.12"]
enable_resolves = true
default_resolve = "default"

[python.resolves]
default = "3rdparty/python/default.lock"
mypy = "3rdparty/python/mypy.lock"
old_app = "3rdparty/python/old_app.lock"
grpc_client = "3rdparty/python/grpc_client.lock"

[source]
root_patterns = [
    "/src/python/*",
    "/src/protos",
]

[black]
config = "build-support/pyproject.toml"

[isort]
config = ["build-support/pyproject.toml"]

[pylint]
config = "build-support/pylint.config"
EOF

Format codes and Apply linter

Format codes.

pants fmt ::

Apply linter.

pants lint ::

Add mypy

"pants.backend.python.typecheck.mypy",

cat <<EOF > pants.toml
[GLOBAL]
pants_version = "2.17.0"

backend_packages = [
    "pants.backend.python",
    "pants.backend.python.lint.bandit",
    "pants.backend.python.lint.black",
    "pants.backend.python.lint.docformatter",
    "pants.backend.docker.lint.hadolint",
    "pants.backend.python.lint.isort",
    "pants.backend.python.lint.pylint",
    "pants.backend.python.lint.pyupgrade",
    "pants.backend.experimental.python.lint.ruff",
    "pants.backend.python.typecheck.mypy",
    "pants.backend.codegen.protobuf.python",
    "pants.backend.docker",
]

[python]
interpreter_constraints = [">=3.11,<3.12"]
enable_resolves = true
default_resolve = "default"

[python.resolves]
default = "3rdparty/python/default.lock"
mypy = "3rdparty/python/mypy.lock"
old_app = "3rdparty/python/old_app.lock"
grpc_client = "3rdparty/python/grpc_client.lock"

[source]
root_patterns = [
    "/src/python/*",
    "/src/protos",
]

[python-protobuf]
mypy_plugin = true


[black]
config = "build-support/pyproject.toml"

[isort]
config = ["build-support/pyproject.toml"]

[pylint]
config = "build-support/pylint.config"

[mypy]
config = "build-support/pyproject.toml"
install_from_resolve = "mypy"
requirements = ["//3rdparty/python:mypy"]
EOF

Write requirements.txt for mypy.

cat <<EOF >3rdparty/python/mypy-requirements.txt
mypy==1.6.1
mypy-extensions==1.0.0
tomli==2.0.1
typing_extensions==4.8.0
EOF

Write BUILD fild for mypy.

cat <<EOF > 3rdparty/python/BUILD
python_requirements(
    name="mypy",
    source="mypy-requirements.txt",
    resolve="mypy",
)
EOF

pants generate-lockfiles ::

Generate source files from proto files.

pants export-codegen ::

Check if *.pyi files are generated.

ls dist/codegen/src/protos/helloworld/v1/

Check mypy

Check mypy.

pants check ::

Output:

Partition #1 - default, ['CPython<3.12,>=3.11']:
src/protos/helloworld/v1/helloworld_pb2.pyi:19: error: Library stubs not installed for "google.protobuf.descriptor"  [import-untyped]
src/protos/helloworld/v1/helloworld_pb2.pyi:19: note: Hint: "python3 -m pip install types-protobuf"
src/protos/helloworld/v1/helloworld_pb2.pyi:19: note: (or run "mypy --install-types" to install all missing stub packages)
src/protos/helloworld/v1/helloworld_pb2.pyi:19: error: Library stubs not installed for "google.protobuf"  [import-untyped]
src/protos/helloworld/v1/helloworld_pb2.pyi:19: error: Skipping analyzing "google": module is installed, but missing library stubs or py.typed marker  [import-untyped]
src/protos/helloworld/v1/helloworld_pb2.pyi:20: error: Library stubs not installed for "google.protobuf.message"  [import-untyped]
src/protos/helloworld/v1/helloworld_pb2_grpc.pyi:19: error: Skipping analyzing "grpc": module is installed, but missing library stubs or py.typed marker  [import-untyped]
src/python/grpc-server/grpc_server/main.py:3: error: Skipping analyzing "grpc": module is installed, but missing library stubs or py.typed marker  [import-untyped]
src/python/grpc-server/grpc_server/main.py:3: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
Found 6 errors in 3 files (checked 8 source files)

Partition #2 - grpc_client, ['CPython<3.12,>=3.11']:
src/protos/helloworld/v1/helloworld_pb2.pyi:19: error: Library stubs not installed for "google.protobuf.descriptor"  [import-untyped]
src/protos/helloworld/v1/helloworld_pb2.pyi:19: note: Hint: "python3 -m pip install types-protobuf"
src/protos/helloworld/v1/helloworld_pb2.pyi:19: note: (or run "mypy --install-types" to install all missing stub packages)
src/protos/helloworld/v1/helloworld_pb2.pyi:19: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
src/protos/helloworld/v1/helloworld_pb2.pyi:19: error: Library stubs not installed for "google.protobuf"  [import-untyped]
src/protos/helloworld/v1/helloworld_pb2.pyi:19: error: Skipping analyzing "google": module is installed, but missing library stubs or py.typed marker  [import-untyped]
src/protos/helloworld/v1/helloworld_pb2.pyi:20: error: Library stubs not installed for "google.protobuf.message"  [import-untyped]
Found 4 errors in 1 file (checked 2 source files)

Partition #3 - old_app, ['CPython<3.12,>=3.11']:
src/python/old-app/old_app/util/util.py:1: error: Skipping analyzing "pendulum": module is installed, but missing library stubs or py.typed marker  [import-untyped]
src/python/old-app/old_app/util/util.py:1: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
Found 1 error in 1 file (checked 3 source files)

Some errors occurred.

Currently, the purpose is not to fix the source code. Ignore these errors.

Update build-support/pyproject.toml config file.

cat <<EOF > build-support/pyproject.toml
[tool.isort]
profile = "black"
line_length = 100

[tool.black]
line-length = 100

[tool.mypy]
ignore_missing_imports = true

[[tool.mypy.overrides]]
module = ['helloworld.v1.*']
ignore_errors = true
EOF

Check mypy again.

pants check ::

19:06:00.61 [INFO] Completed: Typecheck using MyPy - mypy - mypy succeeded.
Partition #1 - default, ['CPython<3.12,>=3.11']:
Success: no issues found in 8 source files

Partition #2 - grpc_client, ['CPython<3.12,>=3.11']:
Success: no issues found in 2 source files

Partition #3 - old_app, ['CPython<3.12,>=3.11']:
Success: no issues found in 3 source files

Passed!

9. Test

Add test code

mkdir -p src/python/console-app/tests/util

cat <<EOF > src/python/console-app/tests/util/util_test.py
from console_app.util.util import now
import pendulum


def test_now():
    x = now()
    assert isinstance(x, pendulum.DateTime)
EOF

Add the below codes to pants.toml file.

[test]
use_coverage = true

[coverage-py]
report = "xml"

Update pants.toml

cat <<EOF > pants.toml
[GLOBAL]
pants_version = "2.17.0"

backend_packages = [
    "pants.backend.python",
    "pants.backend.python.lint.bandit",
    "pants.backend.python.lint.black",
    "pants.backend.python.lint.docformatter",
    "pants.backend.docker.lint.hadolint",
    "pants.backend.python.lint.isort",
    "pants.backend.python.lint.pylint",
    "pants.backend.python.lint.pyupgrade",
    "pants.backend.experimental.python.lint.ruff",
    "pants.backend.python.typecheck.mypy",
    "pants.backend.codegen.protobuf.python",
    "pants.backend.docker",
]

[python]
interpreter_constraints = [">=3.11,<3.12"]
enable_resolves = true
default_resolve = "default"

[python.resolves]
default = "3rdparty/python/default.lock"
mypy = "3rdparty/python/mypy.lock"
pytext = "3rdparty/python/pytest.lock"
old_app = "3rdparty/python/old_app.lock"
grpc_client = "3rdparty/python/grpc_client.lock"

[source]
root_patterns = [
    "/src/python/*",
    "/src/protos",
]

[test]
use_coverage = true

[coverage-py]
report = "xml"

[black]
config = "build-support/pyproject.toml"

[isort]
config = ["build-support/pyproject.toml"]

[pylint]
config = "build-support/pylint.config"

[python-protobuf]
mypy_plugin = true

[mypy]
config = "build-support/pyproject.toml"
install_from_resolve = "mypy"
requirements = ["//3rdparty/python:mypy"]
EOF

Test

pants tailor ::
pants test ::

Output:

Name                                              Stmts   Miss  Cover
---------------------------------------------------------------------
src/python/console-app/console_app/__init__.py        0      0   100%
src/python/console-app/console_app/util/util.py       3      0   100%
src/python/console-app/tests/__init__.py              0      0   100%
src/python/console-app/tests/util/util_test.py        5      0   100%
---------------------------------------------------------------------
TOTAL                                                 8      0   100%


Wrote xml coverage report to `dist/coverage/python`

Check if a report file is created.

ls dist/coverage/python

Create a new directory

Install pants

1. First application

Create console-app using poetry

Write source codes

Create pants.toml

Create BUILD files using pants tailor command

Run application via source file

Run application via binary file

2. gRPC server

Create grpc-server using poetry.

Create proto file.

Update configurations.

Write initial source codes

Update source codes

Run gRPC server

Create Docker image

3. Third application

Check dependencies of console-app

Create console2-app using poetry

Write source codes

Run application

Check dependencies of console-app again

Run application again

4. Fourth application

Create old-app using poetry.

Write source codes

5. gRPC client

Create grpc-client using poetry.

Write source codes

Run gRPC server.

Run gRPC client

6. Use Lockfiles

Generate lockfiles

Generate lockfiles for old_app and others

7. Generating Source code from proto files for each resolve

Add grpc_client resolve

Add another protobuf_sources to BUILD file for proto files.

Update BUILD files for grpc-client application

Update pants.toml for grpc-client application

8. Linters and formatters

Add bandit

Add other linters

Format codes and Apply linter

Add mypy

Check mypy

9. Test

Add test code

Update pants.toml

Test

Discussion

Install `pants`

Create `console-app` using poetry

Create `pants.toml`

Create `BUILD` files using `pants tailor` command

Create `grpc-server` using poetry.

Check dependencies of `console-app`

Create `console2-app` using poetry

Check dependencies of `console-app` again

Create `old-app` using poetry.

Create `grpc-client` using poetry.

Add `grpc_client` resolve

Add another `protobuf_sources` to BUILD file for proto files.