GitLab Runner and GitLab Pages - s7/7

日本語記事準備中-シリーズ前後記事リンク追加予定

first post: Cheap Home LAN Playground Using Docker

This is the seventh post in the series. Now I have GitLab server to play with, I am going to setup GitLab Runner and GitLab Pages and explain the steps involved in this post.

As a quick introduction, GitLab Runner is something that runs jobs based on the instruction available on GitLab project repository, and GitLab Pages is where GitLab users can use to host websites. What I can do with these is that I can commit/push website content to a project, let GitLab Runner process it and generate website files and upload them to GitLab Pages to host them.

GitLab Runner

https://docs.gitlab.com/runner/

GitLab Runner is an application that runs outside GitLab, and it works with GitLab by periodically checking in to see if there is any job to execute.

I will run this GitLab Runner using docker compose as I have been doing for others so far.

Creating a group and obtaining registration token

GitLab will not let any anonymous Runner to just come and read the project content hosted. GitLab uses token to let GitLab Runner identify itself to come read the project content.

I am going to create a group and then retrieve a token. This way, when I use that token to register GitLab Runner, it is going to be available for the projects under that group.

https://docs.gitlab.com/ee/ci/runners/runners_scope.html#group-runners

Let me create a new group named "pages", and then navigate to the Runner menu as described in the link above to find the token string for a GitLab Runner to register and work with this group.

GitLab Runner registration

I will run mere docker run to run GitLab Runner container to complete the registration. When the container tries to communicate with the GitLab server, it needs to trust the self-signed certificate it's using.

https://docs.gitlab.com/runner/configuration/tls-self-signed.html

# as always, creating a directory for GitLab Runner
mkdir $HOME/mylan/runner
cd $HOME/mylan/runner

# store the certificate in $HOME/mylan/runner/gitlab.mylan.local.crt
openssl s_client -showcerts -connect gitlab.mylan.local:443 -servername gitlab.mylan.local < /dev/null 2>/dev/null | openssl x509 -outform PEM > gitlab.mylan.local.crt

https://docs.gitlab.com/runner/register/index.html#docker

Now here is the docker run command to run. The output says the registration was successful.

• config will be stored on the host machine's $HOME/mylan/runner/config directory
• the certificate obtained will be on the container machine as /tmp/gitlab.mylan.local.crt
• specifying the GitLab server in --url "https://gitlab.mylan.local/"
  • --clone-url is also needed, otherwise the runner tries to access the project repository using plain http
• registration token obtained previously must be used in --registration-token "token_here"
• --tls-ca-file=/tmp/gitlab.mylan.local.crt will make this GitLab Runner container trust the GitLab server using self-signed certificate

$ docker run --rm -v "$(pwd)"/config:/etc/gitlab-runner \
>   -v "$(pwd)"/gitlab.mylan.local.crt:/tmp/gitlab.mylan.local.crt \
>   gitlab/gitlab-runner register \
>   --non-interactive \
>   --executor "docker" \
>   --docker-image alpine:latest \
>   --url "https://gitlab.mylan.local/" \
>   --clone-url "https://gitlab.mylan.local/" \
>   --registration-token "GR1348941H2DxBfrLkMrFDfFizvfk" \
>   --run-untagged="true" \
>   --locked="false" \
>   --access-level="not_protected" \
>   --tls-ca-file=/tmp/gitlab.mylan.local.crt
Runtime platform                                    arch=amd64 os=linux pid=7 revision=7178588d version=15.5.1
Running in system-mode.

Registering runner... succeeded                     runner=GR1348941H2DxBfrL
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

Configuration (with the authentication token) was saved in "/etc/gitlab-runner/config.toml"

I can see on GitLab server GUI that the runner is registered.

Running GitLab Runner using Docker Compose

Now create docker-compose.yml file for the GitLab Runner and run it with docker compose up -d.

services:
  runner:
    image: 'gitlab/gitlab-runner:alpine3.15-v15.3.3'
    restart: always
    container_name: mylan-runner
    volumes:
      - type: bind
        source: ./config
        target: /etc/gitlab-runner
      - type: bind
        source: ./gitlab.mylan.local.crt
        target: /tmp/gitlab.mylan.local.crt
      - type: bind
        source: /var/run/docker.sock
        target: /var/run/docker.sock

https://hub.docker.com/r/gitlab/gitlab-runner

I can see that the runner is in online status.

Testing out GitLab Runner

Let me test out this runner following the tutorial available in the official documentation.

https://docs.gitlab.com/ee/ci/quick_start/

Create a test project

Create a new project named "runner test" under the "pages" group.

I will clone the project repository on my first machine and move to the repository directory.

# clone the repository at home dir
cd $HOME
git clone https://gitlab.mylan.local/pages/runner-test.git
cd runner-test

Add .gitlab-ci.yml file

Create .gitlab-ci.yml file at $HOME/runner-test/.gitlab-ci.yml and push the change to the GitLab server.

Here is the content.

build-job:
  stage: build
  script:
    - echo "Hello, $GITLAB_USER_LOGIN!"

test-job1:
  stage: test
  script:
    - echo "This job tests something"

test-job2:
  stage: test
  script:
    - echo "This job tests something, but takes more time than test-job1."
    - echo "After the echo commands complete, it runs the sleep command for 20 seconds"
    - echo "which simulates a test that runs 20 seconds longer than test-job1"
    - sleep 20

deploy-prod:
  stage: deploy
  script:
    - echo "This job deploys something from the $CI_COMMIT_BRANCH branch."
  environment: production

Once I create the new file .gitlab-ci.yml, git status shows that there is an untracked file. I do git add, git commit, and git push to update the project repository.

❯ git status
On branch main
Your branch is up to date with 'origin/main'.

Untracked files:
  (use "git add <file>..." to include in what will be committed)
	.gitlab-ci.yml

nothing added to commit but untracked files present (use "git add" to track)
❯ git add .gitlab-ci.yml
❯ git commit -m "Add tutorial cicd file"
[main afe0df9] Add tutorial cicd file
 1 file changed, 23 insertions(+)
 create mode 100644 .gitlab-ci.yml
❯ git push
Enumerating objects: 4, done.
Counting objects: 100% (4/4), done.
Delta compression using up to 4 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 583 bytes | 583.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
To https://gitlab.mylan.local/pages/runner-test.git
   3241e74..afe0df9  main -> main

When I go check the GitLab GUI, I can see that the jobs were executed. GitLab Runner is continuously access GitLab to see if there is anything to do, finds the updated repository with .gitlab-ci.yml file which is job instructions, and executes them.

Here is the capture of "build-job" and I can see it's executing echo "Hello, $GITLAB_USER_LOGIN!" which is "Hello, ghost!".

GitLab Pages

You can find numerous things you can do with GitLab CI/CD in the official document. One I would like to cover is GitLab Pages. As mentioned in the beginning of the post, I will update contents to publish in a project repository, have GitLab Runner run programs to generate web contents, and give them to GitLab Pages to publish.

https://docs.gitlab.com/ee/ci/examples/

https://docs.gitlab.com/ee/user/project/pages/index.html

Setting up GitLab Pages

https://docs.gitlab.com/ee/administration/pages/index.html

I need to enable GitLab Pages and specify its base URL. I can do so by editing gitlab.rb configuration file for GitLab. In my setup, the file is located at $HOME/mylan/gitlab/config/gitlab.rb.

https://docs.gitlab.com/ee/user/project/pages/getting_started_part_one.html#gitlab-pages-default-domain-names

As described in the link above, the URL will be in this format: http(s)://groupname.example.io/projectname. I have created a group named "pages" and also a project named "runner-test", so I would like the Pages URL to be https://pages.mylan.local/runner-test/.

I picked up part of GitLab Pages configuration section from the gitlab.rb configuration file below. I changed these two lines, pages_external_url and gitlab_pages['enable'].

################################################################################
## GitLab Pages
##! Docs: https://docs.gitlab.com/ee/administration/pages/
################################################################################

##! Define to enable GitLab Pages
pages_external_url "http://mylan.local/"
gitlab_pages['enable'] = true

Editing the configuration file will not immediately take effect and change the GitLab server. For example in the previous Nginx configuration changes, I restarted the server, or the docker container running Nginx. In case of GitLab, I can just run gitlab-ctl reconfigure to have GitLab reconfigure and update itself.

# this runs in foreground and you will see 
# all the GitLab logs generated while it reconfigures itself
docker exec gitlab gitlab-ctl reconfigure

# or, add "-d" option to let it run in the background
docker exec -d gitlab gitlab-ctl reconfigure

Testing out GitLab Pages with plain html

Let me test out GitLab Pages by using the example project available on GitLab.

https://gitlab.com/pages/plain-html

I will take the similar steps as I did to test out GitLab Runner. I create a new project "plain-html" under the "pages" group, create html and css files under /public directory, and then create .gitlab-ci.yml.

So here I have the new project, added "/public" directory containing "/public/index.html" and "/public/style.css" files, and also ".gitlab-ci.yml" file.

Here is the .gitlab-ci.yml file content.

# This file is a template, and might need editing before it works on your project.
# Full project: https://gitlab.com/pages/plain-html
image: busybox
pages:
  stage: deploy
  script:
    - echo "The site will be deployed to $CI_PAGES_URL"
  artifacts:
    paths:
      - public
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

Now what happens is that the GitLab Runner finds a new commit with CI/CD instructions file .gitlab-ci.yml and figures out that there are jobs to execute. I can see on the project CI/CD menu that there is a new job being executed.

Add DNS record

Now, before I access this GitLab Pages URL https://pages.mylan.local/plain-http, let me quickly add the new DNS record in my local Unbound DNS running on Docker.

On my first machine running Unbound DNS docker container, I'd go edit $HOME/mylan/dns/config/a-records.conf to add this new line local-data: "pages.mylan.local. IN A 192.168.1.56" and restart the container.

cd $HOME/mylan/dns
# edit config/a-records.conf to add a new record for pages.mylan.local
docker compose restart

Add web server

I also need the additional reverse proxy web server just like the one I created enable access to https://gitlab.mylan.local.

cd $HOME/mylan/rp/conf.d
cp gitlab.conf pages.conf
# edit pages.conf accordingly
# just changing "server_name" line as below will suffice
### server_name pages.mylan.local;

# once the new file is ready, restart the server
cd $HOME/mylan/rp
docker compose restart rp

Access the Pages site

GitLab Pages is now enabled, web contents were uploaded by GitLab Runner as instructed in the .gitlab-ci.yml file, DNS record and reverse proxy for pages.mylan.local is both ready, and all is set! I can now access the URL and see below website.

GitLab Pages and Mkdocs

The first example requires me to prepare all the html, css, and other files to be published on GitLab Pages. In this and the following sections, I will have GitLab Runner execute a static site generator program to prepare html and all the other files that builds up a website.

I am going to use Mkdocs in this section.

https://www.mkdocs.org/

Let me start this by creating a new project named "mkdocs" under the existing "pages" group, and start placing required files. I can see in the getting started page in the Mkdocs documentation that the minimal setup is to have mkdocs.yml file and docs directory with markdown files, but in this section let me go directly with what I ended up with including the customization and plugins.

Here is the /mkdocs.yml file.

site_name: Mkdocs and GitLab Pages
site_url: https://pages.mylan.local/mkdocs
site_description:
  "Mkdocs and GitLab Pages"
site_dir: public
theme:
  palette: # https://squidfunk.github.io/mkdocs-material/setup/changing-the-colors/
    scheme: default
    primary: green
    accent: blue
  name: material
  features:
    - tabs
    - navigation.top
  font:
    text: Noto Sans
    code: Oxygen Mono
markdown_extensions:
  - toc:
      permalink: true
      toc_depth: 3
plugins:
  - search
  - git-revision-date
  - awesome-pages
  - mermaid2:
      arguments:
        theme: 'forest'
extra_javascript:
    - https://unpkg.com/mermaid/dist/mermaid.min.js

Here is /requirements.txt file. This is rather for GitLab Runner to use. Mkdocs runs in python, so in CI/CD job, the runner will use python image to install these packages, and then run Mkdocs to generate website contents to upload on GitLab Pages.

# Mkdocs
mkdocs

# Theme
mkdocs-material

# Plugins
mkdocs-git-revision-date-plugin
mkdocs-mermaid2-plugin
mkdocs-awesome-pages-plugin

Now add /docs directory and I can add markdown files and subdirectories however I like. Below tree output is what I ended up with. I will just cover one md filie later in the section along with a screenshot.

The .pages file is for Mkdocs to use. It just contains one line telling Mkdocs what to label this folder.

You may notice the numbering on directories and files under /docs. I am doing so so that the markdown_extensions automatically generates TOC and places contents in order.

https://www.mkdocs.org/user-guide/configuration/#markdown_extensions

❯ tree
.
 |-requirements.txt
 |-.git
 |-docs
 | |-index.md
 | |-20_placeholder
 | | |-.pages
 | | |-100.md
 | |-10_blog
 | | |-.pages
 | | |-105_posts.md
 | | |-100_posts.md
 |-README.md
 |-mkdocs.yml

Now I add .gitlab-ci.yml file. There is an example GitLab project for Mkdocs in the link below, and has the .gitlab-ci.yml example I can just copy and use. I have removed the python image tag so that the latest version available will be used.

https://gitlab.com/pages/mkdocs

image: python

before_script:
  - pip install -r requirements.txt

test:
  stage: test
  script:
  - mkdocs build --strict --verbose --site-dir test
  artifacts:
    paths:
    - test
  rules:
    - if: $CI_COMMIT_REF_NAME != $CI_DEFAULT_BRANCH

pages:
  stage: deploy
  script:
  - mkdocs build --strict --verbose
  artifacts:
    paths:
    - public
  rules:
    - if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH

Once the job gets executed, I can access https://pages.mylan.local/mkdocs/ to see the website generated using Mkdocs.

Here is how the generated website looks like. I can see the TOC in the left pane. The title of each file come from the h1 text, and the folder title comes from what's written in the .pages file. I can also see that the Mermaid is working fine, which is an external plugin generating diagram from text.

I am not going through every single files, but let me just share .pages file and the original markdown file from the above capture as example.

So here are the .pages files.

❯ cat docs/10_blog/.pages
title: Blog
❯ cat docs/20_placeholder/.pages
title: Another section - placeholder

And the markdown file.

❯ cat docs/10_blog/105_posts.md
# Second file

This is the second file.

# Another h1

Hello there.

## Mermaid

> https://mermaid-js.github.io/mermaid/

```mermaid
graph LR

    user -- http --> jupyter(jupyter.mylan.net:8888)

```

Closing

This post basically concludes the series. I would also like to introduce "mdBooks", one similar to MkDocs, and "marp" which can generate slideshow served as a web page, so I might add a couple more short posts after this.

I have been working as a network engineer for seven years. The different roles I have played over these years provided me with an exposure to very rich variety of technologies and products. They were not just routers and switches, but also DDI (DNS, DHCP, IPAM), loadbalancer, GSLB, firewall, Internet proxy, and etc., and playing around with Docker enriched my experience to setup and manage different kind of services.

There were also many occassion at work where I wrote script to automate jobs, study data gathered, and generate reports. Studying git was my original purpose of running my own GitLab server, but it actually facilitated me writing more in private, not just codes but blogs, stories, documents, recipes, or anything. Getting to know of CI/CD, GitLab Pages, and static site generators was also a great addition.

Thus, I thought having your own playground using Docker worth a tons, and I hope this blog series to be of a good entry point to some of the readers. Please share this blog series with your friends, colleagues, who is interested in start trying out Docker and running services at their home LAN.