iTranslated by AI
Installing SML# (smlsharp) on Apple Silicon Mac
This article is for day 25 of the ML (Meta Languages) Advent Calendar 2023.
SML# is a programming language implementation that adds several extensions, including record polymorphism, to Standard ML.
However, as of now (late 2023), it only supports x86_64 Unix-like OSs and does not support AArch64 environments like Apple Silicon Macs.
Since SML# uses LLVM for code generation, porting it to other architectures should not be difficult. In fact, I (mod_poppo) worked on the porting about a year ago and confirmed that simple programs run (complex programs have not been tested).
In this article, I will explain how others can also run SML# on an Apple Silicon Mac.
Preparation
This guide assumes the use of Homebrew. Install LLVM 13, GMP, and libffi. Other tools might be required, so please use your discretion.
$ brew install llvm@13 gmp libffi
You can check the installation path of the packages with brew --prefix llvm@13, brew --prefix gmp, or brew --prefix libffi.
Adjust the number of cores in make -j8 in the following steps according to your machine. I am using an 8-core Apple M1, so I specify 8.
According to the documentation, the recommended LLVM version is 11.1.0. However, I used LLVM 13 when I tried it before, and trying a different version feels risky (or tedious), so I will stick with LLVM 13.
Let the working directory be $WORKDIR. Create it wherever you like. A path without spaces is preferable. Example:
$ WORKDIR=$HOME/smlsharp-work
$ mkdir -p $WORKDIR
The final installation destination for SML# (and MassiveThreads) will be $PREFIX_AARCH64. Here, we use /opt/smlsharp. Only AArch64 native binaries will be placed under $PREFIX_AARCH64.
$ PREFIX_AARCH64=/opt/smlsharp
Building MassiveThreads for AArch64
MassiveThreads is a lightweight thread library used by SML#. Mr. Ueno from the SML# development team added AArch64 support around 2021, and the patch seems to have been merged, but since it hasn't been released yet, we will fetch the source and build it ourselves.
$ cd $WORKDIR
$ git clone https://github.com/massivethreads/massivethreads.git
$ cd massivethreads
$ mkdir build-aarch64 && cd build-aarch64
$ ../configure --prefix="$PREFIX_AARCH64" CFLAGS="-Wall -O3"
$ make -j8
$ sudo make install
Method for Those Who Trust Me (mod_poppo)
The SML# compiler is written in SML#. In other words, to build SML#, you need an SML# compiler. Since the compiler description uses SML# extensions, it cannot be compiled with a regular Standard ML compiler.
However, SML# is designed to be buildable with the make command even in environments where SML# is not present. This works because the source archive includes SML# compiled into LLVM IR. Specifically, precompiled/x86_64.ll.xz is used for this purpose.
So, if someone prepares an aarch64.ll.xz, others can use it to build SML#. Therefore, I have prepared a branch that includes aarch64.ll.xz at support-aarch64-precompiled.
However, since it is not realistic for a human to check the contents of aarch64.ll.xz to ensure it doesn't contain malicious code, this method is only for those who can trust the creator of aarch64.ll.xz. So, please use this method if you trust me (mod_poppo).
$ cd $WORKDIR
$ git clone -b support-aarch64-precompiled https://github.com/minoki/smlsharp.git smlsharp-precompiled
$ cd smlsharp-precompiled
$ ./configure --prefix="$PREFIX_AARCH64" --with-llvm="$(brew --prefix llvm@13)" --with-massivethreads="$PREFIX_AARCH64" CFLAGS="-I$(brew --prefix gmp)/include -I$(brew --prefix libffi)/include" LDFLAGS="-L$(brew --prefix gmp)/lib -L$(brew --prefix libffi)/lib"
$ make -j8
$ file src/compiler/smlsharp
src/compiler/smlsharp: Mach-O 64-bit executable arm64
It built successfully. Let's check the operation.
$ src/compiler/smlsharp -Bsrc
SML# 4.0.1-35.gfd6a5036 (2023-12-19 22:12:58 JST) for arm64-apple-darwin22.6.0 with LLVM 13.0.1
# 1 + 1;
val it = 2 : int
# ^D
Looks good. Let's install it.
$ sudo make install
$ export PATH="$PREFIX_AARCH64/bin:$PATH"
$ smlsharp
SML# 4.0.1-35.gfd6a5036 (2023-12-19 22:12:58 JST) for arm64-apple-darwin22.6.0 with LLVM 13.0.1
# 1 + 1;
val it = 2 : int
# fun fact 0 = 1 | fact n = n * fact (n - 1) : IntInf.int;
val fact = fn : intInf -> intInf
# fact 100;
val it =
93326215443944152681699238856266700490715968264381621468592963895217599993229915608941463976156518286253697920827223758251185210916864000000000000000000000000
: intInf
# ^D
Looking nice.
Method to Build Everything Yourself
If you do not trust the aarch64.ll.xz that I created, you will have to start from the x86_64.ll.xz provided by the SML# development team. Fortunately, Macs have Rosetta 2, so you can run x86_64 binaries as well. For Linux users, using QEMU might work.
Note that using the x86_64 version of Homebrew would make installing dependencies easier, but I do not want to install the x86_64 version of Homebrew on an Apple Silicon Mac, so I will prepare the dependencies manually. MacPorts supports Universal Binaries, so using MacPorts might also be an option.
We will install the x86_64 versions of the dependency libraries under $PREFIX_X86_64. Since these will be unnecessary once we are finished, we will prepare them under the working directory.
$ PREFIX_X86_64=$WORKDIR/prefix-x86_64
Checking out SML#
From this point, we will build SML# twice: once for x86_64 and once for AArch64. While separating the source and build directories would allow us to finish with a single clone, SML# does not seem to support out-of-place builds, so we will prepare two copies of the source tree. If your internet connection is slow and git clone is time-consuming, you can clone it locally once and then clone that again.
$ cd $WORKDIR
$ git clone -b support-aarch64 https://github.com/minoki/smlsharp.git
Building GMP for x86_64
As mentioned earlier, I do not want to install the x86_64 version of Homebrew, so I will build GMP myself.
In my environment, make check failed with the default settings, so I am specifying --disable-assembly during configuration.
$ cd $WORKDIR
$ curl -LO https://gmplib.org/download/gmp/gmp-6.3.0.tar.xz
$ tar xf gmp-6.3.0.tar.xz
$ cd gmp-6.3.0
$ arch -x86_64 ./configure --prefix="$PREFIX_X86_64" --disable-assembly
$ arch -x86_64 make -j8
$ arch -x86_64 make check -j8 # At your discretion
$ arch -x86_64 make install
Building libffi for x86_64
Original SML# does not depend on libffi, but my modified support-aarch64 branch does, so we need to prepare libffi.
$ cd $WORKDIR
$ curl -LO https://github.com/libffi/libffi/releases/download/v3.4.4/libffi-3.4.4.tar.gz
$ tar xf libffi-3.4.4.tar.gz
$ cd libffi-3.4.4
$ arch -x86_64 ./configure --prefix="$PREFIX_X86_64"
$ arch -x86_64 make -j8
$ arch -x86_64 make install
Preparing LLVM for x86_64
Building LLVM yourself takes an eternity, so we will fetch the prebuilt binaries.
$ cd $WORKDIR
$ curl -LO https://github.com/llvm/llvm-project/releases/download/llvmorg-13.0.1/clang+llvm-13.0.1-x86_64-apple-darwin.tar.xz
$ tar xf clang+llvm-13.0.1-x86_64-apple-darwin.tar.xz
$ LLVM_X86_64_DIR=$WORKDIR/clang+llvm-13.0.1-x86_64-apple-darwin
Building MassiveThreads for x86_64
Build the MassiveThreads we cloned earlier for x86_64.
$ cd $WORKDIR/massivethreads
$ mkdir build-x86_64 && cd build-x86_64
$ arch -x86_64 ../configure --prefix="$PREFIX_X86_64" CFLAGS="-Wall -O3"
$ arch -x86_64 make -j8
$ arch -x86_64 make install
Building SML# for x86_64
Now, let's build it.
$ cd $WORKDIR
$ git clone "$WORKDIR/smlsharp" smlsharp-x86_64
$ cd smlsharp-x86_64
$ arch -x86_64 ./configure --with-llvm="$LLVM_X86_64_DIR" --with-massivethreads="$PREFIX_X86_64" CC=clang CXX=clang++ CFLAGS="-I$PREFIX_X86_64/include" LDFLAGS="-L$PREFIX_X86_64/lib"
$ arch -x86_64 make -j8
$ file src/compiler/smlsharp
src/compiler/smlsharp: Mach-O 64-bit executable x86_64
Even though it's for x86_64, since we've gone through the trouble of building smlsharp, let's verify its operation.
$ src/compiler/smlsharp -Bsrc
SML# 4.0.1-34.ge642f71e (2023-12-19 09:29:51 JST) for x86_64-apple-darwin22.6.0 with LLVM 13.0.1
# 1 + 1;
val it = 2 : int
# ^D
Looks good.
Preparing aarch64.ll.xz
Let's prepare aarch64.ll.xz to be used for bootstrapping. We will use precompile.mk.
Settings like SMLFORMAT_DEP= are necessary to avoid errors such as UserError.ppg.sml is older than UserError.ppg smlformat (I don't fully understand the details).
Since the libc++.dylib included in the LLVM binary downloaded earlier overlaps with the system's libc++ in an annoying way, I am specifying CXX="clang++ -Wl,-rpath,$LLVM_X86_64_DIR/lib". This should ensure that $LLVM_X86_64_DIR/lib/libc++.dylib is used for both linking and execution of src/llvm/main/anonymize.
$ arch -x86_64 make -f precompile.mk -j8 all ARCH=aarch64 SMLLEX_DEP= SMLYACC_DEP= SMLFORMAT_DEP= CXX="clang++ -Wl,-rpath,$LLVM_X86_64_DIR/lib"
Building SML# for AArch64
Build the AArch64 version of SML# using aarch64.ll.xz.
$ cd $WORKDIR
$ git clone "$WORKDIR/smlsharp" smlsharp-aarch64
$ cd smlsharp-aarch64
$ cp ../smlsharp-x86_64/precompiled/aarch64.ll.xz precompiled/
$ ./configure --prefix="$PREFIX_AARCH64" --with-llvm="$(brew --prefix llvm@13)" --with-massivethreads="$PREFIX_AARCH64" CC=clang CXX=clang++ CFLAGS="-I$(brew --prefix gmp)/include -I$(brew --prefix libffi)/include" LDFLAGS="-L$(brew --prefix gmp)/lib -L$(brew --prefix libffi)/lib"
$ make -j8
$ file src/compiler/smlsharp
src/compiler/smlsharp: Mach-O 64-bit executable arm64
Looking nice. Let's verify the operation.
$ src/compiler/smlsharp -Bsrc
SML# 4.0.1-34.ge642f71e (2023-12-19 09:29:51 JST) for arm64-apple-darwin22.6.0 with LLVM 13.0.1
# 42 * 37;
val it = 1554 : int
# ^D
Looks good. Let's install it.
$ sudo make install
$ export PATH=$PREFIX_AARCH64/bin:$PATH
$ smlsharp
SML# 4.0.1-34.ge642f71e (2023-12-19 09:29:51 JST) for arm64-apple-darwin22.6.0 with LLVM 13.0.1
# 1 + 1;
val it = 2 : int
# fun fact 0 = 1 | fact n = n * fact (n - 1) : IntInf.int;
val fact = fn : intInf -> intInf
# fact 100;
val it =
93326215443944152681699238856266700490715968264381621468592963895217599993229915608941463976156518286253697920827223758251185210916864000000000000000000000000
: intInf
# ^D
Looking nice.
Bonus: Method to Build Everything Yourself (Alternative)
I will also introduce a method to build it in a cross-compiler-like fashion.
The steps up to "Building SML# for x86_64" are the same as in "Method to Build Everything Yourself".
Building SML# for AArch64
Speaking of cross-compilers, it's about build/host/target. Here, we set them as follows:
- build=x86_64 (the platform where the smlsharp build runs)
- host=arm64 (the platform where the built smlsharp will run)
- target=arm64 (the platform where the code output by the built smlsharp will run)
$ cd $WORKDIR
$ git clone "$WORKDIR/smlsharp" smlsharp-aarch64-cross
$ cd smlsharp-aarch64-cross
$ arch -x86_64 ./configure --with-llvm="$LLVM_X86_64_DIR" --with-massivethreads="$PREFIX_AARCH64" --with-smlsharp="$WORKDIR/smlsharp-x86_64" --host=arm64-apple-darwin --target=arm64-apple-darwin CC=clang CFLAGS="-arch arm64 -I$(brew --prefix gmp)/include -I$(brew --prefix libffi)/include" CXX=clang++ CXXFLAGS='-arch arm64' LDFLAGS="-arch arm64 -L$(brew --prefix gmp)/lib -L$(brew --prefix libffi)/lib"
$ arch -x86_64 make -j8
$ file src/compiler/smlsharp
src/compiler/smlsharp: Mach-O 64-bit executable arm64
It seems something was created. Let's try running it.
$ src/compiler/smlsharp -Bsrc
SML# 4.0.1-34.ge642f71e (2023-12-19 09:29:51 JST) for x86_64-apple-darwin22.6.0 with LLVM 13.0.1
# 1 + 1;
dynamic link failed: dlsym(0xa26bfa90, _SML_load): symbol not found
# ^D
It doesn't look good. It even says x86_64-apple-darwin.
Actually, it works if you specify the --target option.
$ src/compiler/smlsharp -Bsrc --target=arm64-apple-darwin
SML# 4.0.1-34.ge642f71e (2023-12-19 09:29:51 JST) for arm64-apple-darwin with LLVM 13.0.1
# 1 + 1;
val it = 2 : int
# ^D
Let's also try compiling a file.
$ cat > hello.sml
print "Hello world!\n";
$ cat > hello.smi
_require "basis.smi"
$ src/compiler/smlsharp -Bsrc --target=arm64-apple-darwin -ohello hello.sml
$ ./hello
Hello world!
I have a perfect understanding of Standard ML!
The reason the --target option is necessary is likely because it is incorrectly referencing the x86_64 version of LLVM. Modifying src/config.mk might fix it (untested). Or perhaps it would be better if SML# always explicitly specified --target to LLVM.
Preparing aarch64.ll.xz
Let's prepare aarch64.ll.xz to be used for bootstrapping.
To override the fact that src/config.mk references LLVM for x86_64, we set the LLC, OPT, LLVM_AS, and LLVM_DIS variables.
Since we are using LLVM from Homebrew here, there are no troublesome issues around libc++.
$ make -f precompile.mk -j8 all ARCH=aarch64 LLC="$(brew --prefix llvm@13)/bin/llc" OPT="$(brew --prefix llvm@13)/bin/opt" LLVM_AS="$(brew --prefix llvm@13)/bin/llvm-as" LLVM_DIS="$(brew --prefix llvm@13)/bin/llvm-dis" SMLLEX_DEP= SMLYACC_DEP= SMLFORMAT_DEP=
$ file precompiled/aarch64.ll.xz
precompiled/aarch64.ll.xz: XZ compressed data, checksum CRC64
Looks good. Now, by following the steps in "Building SML# for AArch64" from "Method to Build Everything Yourself", you will have SML# where build=host=target=arm64.
Conclusion
Well then, everyone, have a pleasant SML# life!
SML# is great, but please also check out LunarML, which I am developing. I recently released v0.1.
Discussion