iTranslated by AI
WasmLinux: Running Linux Kernel and BusyBox in the Web Browser (Without Emulation)
"What's the point of running an OS in a web browser?" I've managed to get it working for now, even though I still don't have an answer to that fundamental question...
*** Note: The prompt does not appear even after a command ends. You need to press Enter manually.** (Bug)
WasmLinux is a WebAssembly "native" Linux environment. Both the kernel and userland are WebAssembly modules compiled with the WebAssembly toolchain (converted to C using wasm2c).
Last time, only the kernel was working, but now you can run ifconfig lo up, ping 127.0.0.1, top, or vi in the browser. This is possible because BusyBox is included. However, it still has zero practicality. It's what you'd call a Proof of Concept.
Previous article:
The update this time is that MUSL libc has been ported and BusyBox is now functional. In short, it's now at the stage where you can create apps for WasmLinux that run in this web browser.
I plan to prepare an SDK and distribution eventually. However, maintenance becomes incredibly difficult once it's built...
How is it different from WebVM or container2wasm?
This topic is unavoidable, so unlike my usual articles, I'll address it right at the beginning.
Solutions like WebVM and container2wasm already exist, which aim to run existing applications by emulating an existing CPU along with the kernel on WebAssembly. Well, even Emscripten used to run by directly converting LLVM-IR generated for ARM32 into JavaScript...
Unlike these, WasmLinux does not emulate a CPU; it aims to directly compile both the kernel and userland into WebAssembly. By doing so, I believe it will better highlight what is missing when trying to run standard C/C++ applications on WebAssembly. Presumably.
In terms of performance, honestly, there probably isn't much difference (modern CPUs powerful enough to run a browser are sufficiently fast). However, it might be advantageous in terms of power consumption because the number of JITC stages can be reduced.
- pros: Once ported to WasmLinux, it can run anywhere WebAssembly runs.
- cons: It creates an environment where standard Linux common sense doesn't apply at all, so porting costs are unlikely to ever be zero. -- While BusyBox required no code modifications, apps that don't work in a NOMMU environment will be very difficult.
- Challenges of NOMMU environments: No
fork(onlyvfork), unable to map files or create shared memory withmmap, ... - Challenges of WebAssembly environments: Strict Harvard architecture (instruction sequence not placed in memory), extremely strict adherence to C specifications, not being ELF in the first place, ...
- Challenges of NOMMU environments: No
The "cons" are quite significant... Still, I don't think it's a completely meaningless project... (more on that later)
Additionally, there is WALI, which suggests reducing porting costs by using Linux APIs directly instead of WASI's POSIX-"like" interfaces. However, since WALI ultimately requires Linux to run, I believe it ranks a step below WasmLinux in terms of portability. It's a competitor for the position of "Linux ABI on Wasm."
What I did
I'm a (former) pro after all, so I can do it if I try. I wonder what kind of pro I am, though...
Structure
The general project structure is also described on the demo page https://wasmlinux-demo.pages.dev/, but to summarize based on what I did:
WasmLinux uses a component called the "Runner" to coordinate the kernel, userland, and the external environment. The Runner is a standard C++20 app, and the browser version is built with Emscripten. The Runner's tasks include:
- Implementing things necessary for LKL and userland operation, such as instantiating Wasm, memory allocation, and thread creation.
- Implementing some syscalls—mainly vfork and clone.
- Implementing a pseudo-inetd to accept telnet connections from the outside and provide a console by starting telnetd within the WasmLinux environment.
It fulfills a role like glue code.
Cheating
As shown in the diagram on the demo page, rather than using the browser-side Wasm engine directly, both Linux and BusyBox are first converted to C with wasm2c and then put through Emscripten again. It was just too much trouble to implement setjmp/longjmp myself...
Also, by taking this approach, a single C++ program can run on both a PC and in the browser, which is advantageous for debugging...
If I implemented something similar to WASI-libc, it should be possible to execute directly with the browser-side WebAssembly engine.
Furthermore:
- I removed the memory access bound check added by wasm2c (it's redundant since it's also performed by the browser-side engine)—this affects the .wasm size more than performance. This alone reduced the size from 25MiB to 17MiB, making it possible to host the demo on Cloudflare Pages.
- Emscripten's std::counting_semaphore implementation was somehow incredibly slow and only sped up while the Dev tools were open (a painful situation!), so I replaced it with POSIX semaphores.
Porting Musl libc
I just copy-pasted the missing headers from ARM32, and that was it!
...Well, I did implement dedicated syscall stubs, of course.
Additionally, since pthread_cancel cannot be implemented due to web browser specifications, I have omitted it for now. You cannot send asynchronous signals to Web Workers. I might implement something like forced termination via SIGTERM in the future, though.
Just like the issues I encountered last time, Musl libc also calls functions by arbitrarily increasing the number of arguments (which is undefined behavior), so I've implemented quite a few workarounds for that.
Implementation of user threads (clone)
The clone syscall is implemented on the Runner side. Since I haven't implemented the fork API yet, the Runner currently only supports creating new threads via CLONE_THREAD (such as pthread_create).
APIs like clone or fork cannot be expressed as standard C functions in WebAssembly, so to bring them to WebAssembly, it's necessary to provide high-level implementations for each specific function. Therefore, the kernel-side implementation cannot be used directly.
Implementation of vfork
Since an MMU cannot be implemented in a browser environment, I've implemented vfork instead of fork. This restriction is the same as NOMMU Linux (uClinux). vfork is a strict subset of fork and is easy to implement because it doesn't involve copying the process. However, its specifications are quite subtle, so it was deprecated in the latest POSIX along with things like getcontext and usleep.
vfork is realized using the GCC extension "statement as expression" and setjmp/longjmp.
GCC and Clang's statement expressions extension allows you to write a sequence where the last expression acts as the return value, similar to functional languages. The do{ ... } while(0); idiom is often used in C macros, and this is essentially a version of that which can return a value.
In POSIX, it is acceptable for vfork (well, all APIs declared in unistd.h) to be a C macro [1], so I utilized that:
I defined it as a macro like the one above. The run_to_execve function here is implemented in the Runner. It keeps the userland thread as is but swaps out only the kernel context to continue execution. Once it reaches the execve point:
- On the child process side, it creates a thread and executes the entry point of the executable in the child thread.
- On the parent process side, it restores the kernel context and returns via
longjmp.
That is how it's implemented.
...However, this way of using setjmp is actually considered a violation in C. Storing the return value of setjmp in a variable is not allowed.
Well, I suppose I can pass it through TLS or something.
Implementation of Signal Delivery (Incomplete)
Due to web browser specifications, it is impossible to implement signals that are perfectly POSIX-compliant. Consequently, the current implementation checks for signals before returning the result of a syscall to the application and executes the signal handler if one exists.
This leads to the issue mentioned at the beginning where the prompt does not appear after a command finishes. Essentially, since read is not interrupted by a SIGCHLD signal when a process terminates, the process cannot proceed until the user provides some input to return from the read syscall (presumably).
Since signals cannot be handled using standard LKL features, I've added a dedicated getsignal operation. If the errno returned by a syscall is of the ERESTART variety, calling this from the Runner side clears the signal delivery state for that thread and provides a function pointer to the signal handler.
Implementation of Pseudo-inetd
The console in the initial demo is produced by a telnet client built into the Runner. To allow telnet connections to the WasmLinux system, I decided to include functionality equivalent to inetd within the Runner.
Since the Runner needs to run on both Win32 and Emscripten, I prepared a dedicated I/O abstraction layer (miniio):
- A backend using libuv (the same I/O library as Node.js) for POSIX and Win32.
- A backend dedicated to local communication implemented using pthreads for Emscripten.
By linking the appropriate backend, I achieved support for both a real telnet client and pseudo-communication on Emscripten with a single codebase.
Incidentally, I implemented the libuv backend first, so the initial functional verification was done using PuTTY.
While a standard inetd allows you to specify the process to start via a configuration file, in this pseudo-inetd, the process to be launched is hardcoded.
Implementation of In-Browser Telnet Client
While there are good articles on Zenn about simple Telnet implementations, I prepared my own implementation (minitelnet) using libtelnet.
Minitelnet is designed to output using POSIX-style terminal ioctl, and it displays in xterm.js via the xterm-pty addon. I learned about the existence of xterm-pty from an article about running Ruby in the browser.
Reflections
Unlike the kernel, userland is somewhat expected to just work, so I didn't face much trouble there.
Porting Cost Issue
In designing WasmLinux, I place high importance on the point: "Not imposing maintenance costs on Upstream." While WASI and WALI require fairly aggressive porting, WasmLinux managed to run BusyBox without any source code modifications (though the build process required some fine-tuning). I believe this is a novel aspect of the project.
Unless this point is emphasized, when aiming for a Linux ABI standard or a Linux distribution for Wasm in the future, the question of who will pay the maintenance cost of porting will stand as a major obstacle. After all, projects like Debian/kFreeBSD have disappeared...
While aiming for "Debian/WebAssembly" with WasmLinux might be difficult in the immediate future, it is something I must keep in mind for expansion. Unfortunately, it is hard for software without an ecosystem to be recognized for its value. I want to make WasmLinux more likely to become a "Debian" than Emscripten or WASI.
To that end, a key tagline for WasmLinux, as written on the demo page, is: Write once (by other folks), run anywhere (i want). I design the system so that I can run programs written by others in the places where I want to run them (without much effort).
How to win against emulation?
In the context of utilizing existing apps, WasmLinux faces competition with solutions such as:
- Emulating existing CPUs (x86, RISC-V, etc.)
- Seriously porting onto WASI or Emscripten
The former, in particular, is superior to WasmLinux in all aspects except for resource consumption, so I wonder what can be done...
The most straightforward approach would be to simply accept that WasmLinux's reason for being is for the evolution of WebAssembly. Indeed, by pushing WasmLinux to its limits, I believe it's possible to identify the functionalities required to port existing C/C++ code to WebAssembly.
I'm also considering aiming for a position as an intermediate distribution format for Linux apps. In other words, because we live in an era where three major CPU architectures—x64, arm64, and RISC-V—coexist, shifting away from the previous x64-only era, there might be a demand for a format that can be re-compiled into the target CPU format after distribution as a package.
One might argue that you could just compile it three times and distribute each version... and that's a valid point. However, by using re-conversion from WebAssembly, we can guarantee that memory layouts and other factors are identical. This opens up possibilities for architecture-mixed computing[2], such as resuming a process on ARM that was previously running on x64. For example, how about as an NDK for a platform like Android? That said, Apple used to mandate LLVM-IR (bitcode) delivery for IoT products like Apple Watch and Apple TV but then stopped, so maybe it won't become that popular...
Furthermore, I feel that providing graphics API bindings would be a strong move. Specifically, if I can bring the WebGL C language bindings I created when I ported Unity WebGL natively to WasmLinux, I'll be able to port existing OpenGL ES2 apps using those EGL/OpenGL ES2 stubs. ...Well, you can do that with Emscripten too, but WasmLinux is an environment more like "real" Linux than Emscripten. It's also theoretically possible with an emulator, but I suspect no one is seriously working on that sort of thing.
Next Step
In the previous article, I set the next step as running BusyBox, and this time I have achieved that. As for the next goal, I would like to run a compiler + CMake + LLDB to recreate a microcontroller development environment on the web. However, that is a bit far off, so I'm wondering if I can first get to the point of running vanilla SDL2 (preparing bindings for OpenGL ES2 and OpenAL).
-
Therefore, we can assume the user won't do things like
#undef vforkor call it like(vfork)(). ↩︎ -
While I'm writing this as if I just thought of it, the truth is actually the opposite. WasmLinux originally started from decentralized POSIX kernel research, so I've intentionally chosen a design that would be advantageous for that. ↩︎
Discussion