Database Persistence with LiteStream in a Sidecar Configuration

Introduction

LiteStream is exciting. LiteStream is a system for creating replicas of SQLite databases. By using this, you can instantly back up SQLite databases updated by an application to S3 storage.

https://twitter.com/mattn_jp/status/1525419149316718592

Yes...

Sidecar Configuration using LiteStream

By using LiteStream to restore the database before the container starts and starting replication after the application has launched, you can remove the container from the host at any time.

Below is a docker-compose.yaml that performs a restore to the /data volume and starts replication after the application starts.

docker-compose.yaml

version: '2'
services:

  restore-container:
    image: litestream/litestream
    container_name: restore
    tty: true
    volumes:
      - '/data'
      - './:/opt/litestream'
    entrypoint: /bin/sh
    command: -c "
      rm -f /data/bbs.sqlite &&
      /usr/local/bin/litestream restore -config /opt/litestream/litestream.yaml /data/bbs.sqlite
      "

  app-container:
    image: mattn/entgo-bbs
    container_name: app
    tty: true
    volumes_from:
      - restore-container:rw
    depends_on:
      - 'restore-container'
    ports:
      - 8989:8989
    command:
      - '/go/bin/entgo-bbs'
      - '-dsn'
      - 'file:/data/bbs.sqlite?cache=shared&_fk=1'

  backup-container:
    image: litestream/litestream
    container_name: backup
    tty: true
    depends_on:
      - 'app'
    volumes_from:
      - restore-container:rw
    entrypoint: /bin/sh
    command: -c "
      /usr/local/bin/litestream replicate -config /opt/litestream/litestream.yaml
      "

Role of restore-container

The restore-container deletes the database file once and then performs a restore using LiteStream. This ensures that the latest database is referenced every time it starts.

Role of app-container

Since the application running in app-container uses SQLite, a database file will be automatically generated if it doesn't exist at startup. Also, since this sample uses ent, the schema will be created upon creation.

app-container starts after restore-container.

Role of backup-container

By the time backup-container starts, the database file will already exist, so it will detect changes and perform replication.

Sample

I have prepared a working sample in the following repository.

https://github.com/mattn/litestream-sidecar-example

Copy litestream.yaml.sample to litestream.yaml and update it with your own S3 storage settings.

litestream.yaml

dbs:
  - path: /data/bbs.sqlite
    replicas:
      - type: s3
        endpoint: https://your-minio.example.com
        name: bbs.sqlite
        bucket: entgo-bbs
        path: bbs.sqlite
        region: us-east1
        forcePathStyle: true
        sync-interval: 1s
        access-key-id: your-minio-access-key
        secret-access-key: your-minio-secret-key

After making the changes, start it with the following command.

$ docker compose up

As a test, stop the containers and run the following to completely remove the containers and volumes.

$ docker compose rm -v -f

You should be able to confirm that the data is restored even after running docker compose up again.

Conclusion

I tried persisting a database using LiteStream in a sidecar configuration. While there are various aspects to consider, such as the possibility of incorrect backups being taken in the event of an accident, I felt that it is at a usable level for now.