Addressing the "jni.h: No such file" error in Bazel 7 (bazelisk --bisect)

First draft: 2023-12-24
Hiroyuki Komatsu (@komatsuh:twitter, @komatsuh:bsky)

Overview

After upgrading to Bazel 7, I encountered an error stating that "jni.h" does not exist. In conclusion, starting from Bazel 7, using the --fat_apk_cpu option requires adding --incompatible_enable_android_toolchain_resolution=false.

This article documents the process of identifying the cause and reaching a solution. It is intended for those who are already using Bazel.

Getting "jni.h: No such file or directory" error in Bazel 7

Android builds using Bazel began failing in my GitHub Actions CI. The following error occurred:

java/com/app/jni.cc:15:10: fatal error: jni.h: No such file or directory
   15 | #include <jni.h>
      |          ^~~~~~~
compilation terminated.

Since the error persisted even after rolling back commits, I suspected that an update to the GitHub Actions environment was the cause and investigated.

In my experience, such errors often occur when the Bazel version is updated, so I checked for Bazel-related changes.

I checked the update history of the GitHub Actions environment and confirmed that the Bazel version had been updated to 7.0.0.

https://github.com/actions/runner-images/commit/28be760bba37289f8f7361bbb84c3d0527e804c3

Specifying Bazel Version with Bazelisk

Bazelisk is a tool for switching between multiple versions of Bazel.

https://github.com/bazelbuild/bazelisk

To build while specifying Bazel version 7.0.0, use the following command:

USE_BAZEL_VERSION=7.0.0 bazelisk build (target_name)

I confirmed that Bazel 7.0.0 produced the same error in my local environment, while Bazel 6.0.0 successfully completed the build.

Identifying the Cause with Bazelisk --bisect

Now that I knew the issue was related to the Bazel version, I investigated the root cause and potential solutions. I used bazelisk --bisect to determine which change in Bazel introduced the problem.

Bisecting (bisect) is a technique for finding the specific change in the history that caused a problem by using a binary search. A similar method exists in Git.

https://git-scm.com/docs/git-bisect

To bisect with Bazelisk, run the following command. It finds the point between version 6.0.0 and 7.0.0 where the build status changes.

bazelisk --bisect=6.0.0..7.0.0 build (target_name)

I expected to finish by checking the results, but the bisect stopped midway.

--- Succeeded at 78db9ae9a545a9586dbb02d7831f5302594e01cb
--- Testing with Bazel built at 05bea52ed3159aa5d15d967f5f56fc084a2b6c73, 6 commits remaining...

...

--- Testing with Bazel built at f7946d0107dd75b2f45bcc79b91c016d075a756d, 3 commits remaining...

2023/12/20 11:23:45 Using unreleased version at commit f7946d0107dd75b2f45bcc79b91c016d075a756d
2023/12/20 11:23:45 Downloading https://storage.googleapis.com/bazel-builds/artifacts/centos7/f7946d0107dd75b2f45bcc79b91c016d075a756d/bazel...
2023/12/20 11:23:45 could not run Bazel: could not download Bazel: failed to download bazel: failed to download bazel: HTTP GET https://storage.googleapis.com/bazel-builds/artifacts/centos7/f7946d0107dd75b2f45bcc79b91c016d075a756d/bazel failed with error 404

I haven't investigated the reason, but it seems that some intermediate commits could not be retrieved. Nevertheless, I was able to narrow it down to 6 commits. From here, I tried specifying the known range of commits.

bazelisk --bisect=78db9ae9a545a9586dbb02d7831f5302594e01cb..05bea52ed3159aa5d15d967f5f56fc084a2b6c73 build (target_name)

This time, as expected, I was able to identify the problematic change. Even if the same error occurred, with only 6 commits, checking all of them manually would be feasible.

--- Failed at 05bea52ed3159aa5d15d967f5f56fc084a2b6c73

--- Bisect Result
first bad commit is https://github.com/bazelbuild/bazel/commit/05bea52ed3159aa5d15d967f5f56fc084a2b6c73

Cause and Solution

The cause of the build error was identified as the following change:

https://github.com/bazelbuild/bazel/commit/05bea52ed3159aa5d15d967f5f56fc084a2b6c73

RELNOTES: Enable Platforms and Toolchains for Android. Android projects will need to stop passing the legacy flag --fat_apk_cpu, and instead use --android_platforms using platforms defined with the @platforms//os:android constraint. The https://github.com/bazelbuild/rules_android repository defines four standard Android platforms for projects that use those rules, @rules_android//:armeabi-v7a, @rules_android//:arm64-v8a, @rules_android//:x86, @rules_android//:x86_64.

As described in the change, the default value of the incompatible_enable_android_toolchain_resolution option has changed to true. This appears to be the reason why builds specifying the --fat_apk_cpu option now result in an error.

As a temporary solution, it seems that setting incompatible_enable_android_toolchain_resolution back to false resolves the issue.

USE_BAZEL_VERSION=7.0.0 bazelisk build (target_name) --incompatible_enable_android_toolchain_resolution=false

I verified that the build succeeds as expected with Bazel 7 using the command above. (In my environment, the --fat_apk_cpu option is specified as a default in .bazelrc.)

Note that, as mentioned in the commit message, switching to --android_platforms instead of --fat_apk_cpu is recommended for the future.

Summary

• An error jni.h: No such file or directory occurred in GitHub Actions.
• I confirmed that the Bazel version in the GitHub Actions build environment was updated.
• I identified the responsible change using bazelisk --bisect.
• From the change history, I found that Bazel 7 requires --incompatible_enable_android_toolchain_resolution=false to use the --fat_apk_cpu option.

Specific Examples

GitHub Actions at the time of the error
https://github.com/google/mozc/actions/runs/7243136493

GitHub Actions after the fix
https://github.com/google/mozc/actions/runs/7279025829

Fix commit
https://github.com/google/mozc/commit/68fea8a05c189a7ff166b36b07eac434569f4cd2