Syncing Livebook on Nerves between Device and Local Machine

Problem

Suppose you are running Nerves Livebook on a device and have written some code in Livebook. If you want to manage those results in a Git repository, it is not straightforward because the Livebook files (.livemd files) reside on the device. You need a way to transfer them from the device to your local machine.

Solution

To solve this, let's use SFTP—which is supported by default in the Nerves Livebook configuration—to retrieve the files from the device to your local machine.

(This article was verified on a Mac. Please adapt the instructions for other operating systems accordingly.)

Preparing Nerves Livebook

First, set up Nerves Livebook so that it is functional according to its documentation.

You should be able to SSH into the device using the following command:

$ ssh [e-mail]

Preparing the repository to store your livebooks

Create a Git repository to manage the Livebooks you've created. The directory structure looks like this:

• Clone destination for Nerves Livebook: /Users/antipop/src/github.com/fhunleth/nerves_livebook
• Local repository to store your own Livebooks: /Users/antipop/src/github.com/kentaro/my_nerves_livebooks

Please adapt the paths to your own environment accordingly.

Preparing to handle sensor data

Since we are using Nerves Livebook, you probably want to try retrieving some sensor data. Let's use kentaro/mh_z19_ex, which allows you to handle the MH-Z19 sensor (capable of measuring CO2 concentration) from Elixir.

I won't explain how to connect the sensor to the Raspberry Pi. You can find diagrams at mh-z19 · PyPI.

Add the dependency modules as follows.

diff --git a/mix.exs b/mix.exs
index 316355a..9151cbd 100644
--- a/mix.exs
+++ b/mix.exs
@@ -54,7 +54,7 @@ defmodule NervesLivebook.MixProject do
       {:kino, "~> 0.2.1"},
 
       # Dependencies for all targets except :host
-      {:circuits_uart, "~> 1.3", targets: @all_targets},
+      {:circuits_uart, "~> 1.3", targets: @all_targets, override: true},
       {:circuits_gpio, "~> 0.4", targets: @all_targets},
       {:circuits_i2c, "~> 0.3", targets: @all_targets},
       {:circuits_spi, "~> 0.1", targets: @all_targets},
@@ -74,7 +74,10 @@ defmodule NervesLivebook.MixProject do
       {:nerves_system_bbb, "~> 2.10", runtime: false, targets: :bbb},
       {:nerves_system_osd32mp1, "~> 0.6", runtime: false, targets: :osd32mp1},
       {:nerves_system_x86_64, "~> 1.15", runtime: false, targets: :x86_64},
-      {:nerves_system_npi_imx6ull, "~> 0.2", runtime: false, targets: :npi_imx6ull}
+      {:nerves_system_npi_imx6ull, "~> 0.2", runtime: false, targets: :npi_imx6ull},
+
+      {:mh_z19, "~> 0.1.0"}
     ]
   end

In the configuration above, the modification to circuits_uart was made because it conflicted with the mh_z19 dependency and caused an error, so I set it to override.

After removing the SD card from the device and inserting it into your computer, build the firmware and burn it to the SD card.

$ mix deps.get
$ mix firmware.burn

Reinsert the SD card into the device and start it up.

Creating a livebook

Once the device has started up successfully, you should be able to access it at http://nerves.local/.

Create a new livebook by clicking the "New livebook" button in the top right. On the page that was created, I ran the code to read data from the MH-Z19 as shown below.

It worked!

Don't forget to save the created livebook to a file.

Retrieving files from the device to your local machine

Now, the files created above are located on the device. To commit these files to the Git repository where you manage your livebooks, you need to transfer all the .livemd files from the directory on the device where your livebooks are stored to your local Mac. Let's use SFTP for this purpose.

Create a file named sync-my-livebooks with the following content:

cd /data/livebooks/notebook
get *

Run the following command from the repository where you want to manage your livebooks. You will be prompted for a password; enter nerves as indicated.

$ sftp [e-mail] < sync-my-livebooks
Nerves Livebook
https://github.com/fhunleth/nerves_livebook

ssh [e-mail] # Use password "nerves"

Password:

Then, the contents of sync-my-livebooks will be executed as SFTP commands, allowing you to retrieve the livebook files from the device. After that, you can simply git commit the retrieved files as usual.

Supplement

Why not use SCP?

By default, Nerves replaces the shell with IEx (Elixir's REPL). Additionally, it executes content sent via SSH as Elixir code by default. Therefore, SCP cannot be used in the usual way. Using SFTP is recommended.

For Nerves SSH configuration, please refer to NervesSSH — nerves_ssh v0.2.2.

Why not use SSHFS?

According to Embedded Elixir, it seems possible, but when I tried it, I could not mount it successfully. If SSHFS worked, it would allow for automatic synchronization, which would be ideal.

If you cannot SSH into the device

In this verification, I am using a Raspberry Pi 3 (rpi3), but with the default settings, SSH via Ethernet was not possible. Therefore, I modified the configuration file for rpi3 as follows, built the firmware myself, and applied it to the device.

diff --git a/config/rpi3.exs b/config/rpi3.exs
index c3049d8..d2fbc47 100644
--- a/config/rpi3.exs
+++ b/config/rpi3.exs
@@ -5,6 +5,6 @@ import Config
 config :vintage_net,
   regulatory_domain: "US",
   config: [
-    {"eth0", %{type: VintageNetEthernet, ipv4: %{method: :dhcp}}},
+    {"eth0", %{type: VintageNetDirect}},
     {"wlan0", %{type: VintageNetWiFi}}
   ]

Now, I can SSH via Ethernet.