iTranslated by AI
How to fix "/usr/bin/ld: cannot find -lpython3.9: No such file or directory"
This is a personal memo for myself. Please let me know if I have said anything incorrect.
Background
When I tried to build the example from pola-rs/pyo3-polars in a Python environment prepared with rye:
> make install
...
= note: /usr/bin/ld: cannot find -lpython3.9: No such file or directory
collect2: error: ld returned 1 exit status
error: could not compile `expression_lib` (lib) due to previous error
💥 maturin failed
Caused by: Failed to build a native library through cargo
...
I got this error and was unable to build.
Environment
❯ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
...
❯ python --version
Python 3.9.16
❯ venv/bin/pip list
Package Version
-------------- -------
expression_lib 0.1.0
maturin 1.3.1
pip 22.0.4
polars 0.19.12
setuptools 58.1.0
tomli 2.0.1
Solution
Build using the following command.
> RUSTFLAGS='-L <PATH_TO>/.rye/py/cpython@3.9.16/install/lib' make run
Cause
When maturin (or rather, Rust) dynamically links the Python version being used (in a Linux environment), it looks in /usr/lib/. However, when managing the environment with rye, the libraries do not exist there; they are located inside ~/.rye (unless you customize it yourself). Therefore, it is likely that the error occurred because maturin could not find the library.
You just need to explicitly tell maturin where that location is. In Rust, this is possible by setting RUSTFLAGS='-L <PATH_TO_LD_LIBRARY_PATH>'. Of course, it is also possible by setting LD_LIBRARY_PATH.
By the way, if you are using static linking, I have a feeling you need to set LIBRARY_PATH. Currently, I deal with this by deciding which one is needed after attempting to build and determining whether the required file is dynamic or static.
Additional Notes
When testing polars, there were cases where just setting RUSTFLAGS='-L ... would fail, and LIBRARY_PATH also had to be set. That is:
RUSTFLAGS='-L <PATH_TO>/.rye/py/cpython@3.9.16/install/lib' make test
results in an error, but
LIBRARY_PATH=<PATH_TO>/.rye/py/cpython@3.9.16/install/lib/ RUSTFLAGS='-L <PATH_TO>/.rye/py/cpython@3.9.16/install/lib' make test
works without issues. I haven't been able to track down the exact reason yet, so if anyone knows, please let me know.
memo
This is completely unrelated to the main content, but it seems that you can use the environment variable LD_PRELOAD to override dynamic libraries without recompilation.
Discussion