iTranslated by AI
Goodbye libsparseir, Hello sparse-ir-rs
Introduction
This article is a technical story about migrating libsparseir, a C++ port of the library sparse-ir written in Python and Julia, to sparse-ir-rs, which has been further ported to Rust.
For the process up to the C++ port, please refer to the following article:
There are various reasons why we moved to Rust despite having completed the C++ implementation. Hiroshi Shinaoka (Shinaoka-san) explains this in detail in his article below:
Personally, I was in favor of switching from C++ because Rust has a package manager. While it took me six months to port from SparseIR.jl to C++, with current technology (Cursor), it seems it was possible to Rust-ify it in about two months. Shinaoka-san himself is well-versed in the physics background surrounding sparse-ir, which shows how important domain knowledge is.
sparse-ir-rs is a Rust crate that handles the core implementation of the "intermediate representation (IR) for imaginary-time propagators."
Julia, Python, and Fortran call this implementation via the C-API. In this article, I will talk about that.
Basic Strategy (Calling functions via C-API)
When calling the core implementation written in Rust from other languages, we decided to call the functions via a C-API.
This follows the idea from the era of the C++ implementation, libsparseir.
When creating a LogisticKernel, the C-API spir_logistic_kernel_new is used.
The spir_kernel mentioned in spir_kernel::new_logistic(lambda) is a struct defined as follows:
new_logistic is a method defined below that wraps the Rust implementation of LogisticKernel defined within the sparse-ir crate.
By returning Box::into_raw(Box::new(kernel)), a pointer is passed to the caller (e.g., Python, Julia).
The Rust implementation of LogisticKernel is defined as follows:
To access the lambda held by the struct from other languages, use the C-API spir_kernel_get_lambda.
The result is stored in lambda_out.
Integration from Python
When calling functions via the C-API from Python, we use the ctypes module.
Similar to the example in the link above, it can be called as follows:
import ctypes
_lib = ctypes.CDLL("path/to/sharedlib")
# Kernel functions
_lib.spir_logistic_kernel_new.argtypes = [c_double, POINTER(c_int)]
_lib.spir_logistic_kernel_new.restype = spir_kernel
def logistic_kernel_new(lambda_val):
"""Create a new logistic kernel."""
status = c_int()
kernel = _lib.spir_logistic_kernel_new(lambda_val, byref(status))
if status.value != COMPUTATION_SUCCESS:
raise RuntimeError(f"Failed to create logistic kernel: {status.value}")
return kernel
For the C++ implementation, I used to write spir_logistic_kernel_new.argtypes and spir_logistic_kernel_new.restype manually with the help of AI. Currently, I have it automatically generate argtypes and restype by parsing the header file sparseir.h that corresponds to the Rust C-API.
Furthermore, the "header file sparseir.h corresponding to the Rust C-API" itself can also be automatically generated. This uses a Rust library called cbindgen to create headers for building C-APIs from Rust libraries.
The invocation of cbindgen is controlled in build.rs.
This means developers only need to know the cargo build command. Developers can focus on the C-API design of the Rust implementation and the development of core algorithms.
Integration from Julia
Julia has the Clang.jl package, which automatically generates Julia functions that use ccall from the sparseir.h containing the C-API signatures. While we built the scripts and mechanisms for automatic generation in Python by relying on AI, in Julia, we can just use the features of Clang.jl. In this sense, Julia is slightly easier.
Python Package Build System
Python package management uses uv. For bundling the shared library of the C++ implementation libsparseir, we relied on scikit-build.
In the Rust implementation sparse-ir-rs, we use hatchling as the build backend.
By configuring pyproject.toml as follows:
build-backend = "hatchling.build"
And by writing hatch_build.py, we can now ensure that the Rust project build cargo build --release runs and the shared library is generated when performing operations like uv sync.
I also left this part to the AI. I used Cursor's Auto model, Composer (Opus 4.5), to generate it as needed.
The CI/CD pipeline for registration to PyPI, etc., mostly reuses the resources from libsparseir.
The only change was adding some preprocessing to include the Rust implementation when cibuildwheel runs.
Julia Build System
Building and registering the shared library is done through Yggdrasil, which is the same as it was for C++.
Due to the switch to Rust, the number of supported platforms has decreased, but since Linux environments and Apple Silicon are supported, it shouldn't be an issue within our expected range.
What Changed by Switching from C++ to Rust
Perhaps due to the evolution of AI models, things have become somewhat easier. The error messages from the compiler are easy to understand. While errors occasionally occur when calling Rust resources from Julia, it's now much clearer where the bug is happening. In C++, a segmentation fault would crash the process, but in Rust, by using assert! macros, it's easier to distinguish whether the failure occurred within Rust or on the Julia side.
Being able to build with cargo build was a relief. In C++, I had to create build directories with cmake and maintain a "bonsai" of CMakeLists.txt, but that effort is no longer necessary.
Incidentally, I didn't write the Rust code by hand; I left it all to Cursor's AI agent. While I don't fully understand the fine-grained syntax yet, the AI filled in the details, allowing me to review the code from a high-level perspective. Also, the VS Code Rust extension's feature to run specific tests was invaluable when writing debug code. In C++, I had to add debug files or edit CMakeLists.txt every time I wanted to run a specific test, so the workload has decreased.
To be honest, I don't really want to use C++ for new projects where I want to write high-performance code in a static language. In the current era where generative AI, LLMs, and AI agents are widespread, it feels better to write in modern Rust, which has a package manager.
I hope the combination of Julia + Rust increases even more.
Discussion