The Fastest Way to Build a Website Monitoring System

This article is a migration to Zenn of a post written on another blog on July 3, 2021.

I thought about how to quickly build a monitoring system that periodically accesses a certain website at a reasonable interval (e.g., once every 10 minutes) without causing a DoS, and sends a notification if there are any updates.

---

While there are various options for notification media, I find it easier to respond to Slack than to LINE, Twitter, or Discord, so I decided to use Slack for notifications.
Slack has had "Incoming Webhooks" for a long time, which was easy to use, but these days a proper SDK is also available, so I think that is easier to work with now.

https://qiita.com/seratch/items/8f93fd0bf815b0b1d557

Once you obtain the Slack token as described in the article, set it in an environment variable named SLACK_API_TOKEN and write the difference notification script check.py as follows.

from difflib import context_diff
import os

import requests
from slack_sdk.web import WebClient
client = WebClient(os.environ["SLACK_API_TOKEN"])

# Target link for monitoring
url = "https://XXXXXXXXXXXXXXXXXXXXXXXXXXXXX"

# Read the old file
old_data = open("dump", "r").read().splitlines(keepends=True)
# Remove newline characters
old_data = [line.rstrip() for line in old_data]

# Download the new file
new_data = requests.get(url).text.splitlines(keepends=True)
# Remove newline characters
new_data = [line.rstrip() for line in new_data]


if old_data != new_data:
    out = []
    for line in context_diff(old_data, new_data):
        out.append(line)
    
    client.chat_postMessage(text="".join(out), channel="#ZZZZZZZZZZZZZZ")

    with open("dump", "w") as f:
        f.write("".join(new_data))

I decided to save the site data in a file named dump, so start by creating an empty file with this name.
You can confirm that running this script correctly sends notifications to Slack.

---

Next, let's consider how to run this periodically. If you have a server, you could use crontab, but how can we do it serverless? Here, I will explore GitHub Actions.

Push the check.py and dump file mentioned above into a suitable repository.
Next, prepare a GitHub Actions file as follows.

name: cron

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]
  schedule:
    - cron:  '*/10 0-8 * * *'
    
env:
  SLACK_API_TOKEN: ${{ secrets.SLACK_API_TOKEN }}

jobs:
  build:

    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v2
    - name: Set up Python 3.9
      uses: actions/setup-python@v2
      with:
        python-version: 3.9
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install requests
        pip install slack_sdk
    - name: run checker
      run: |
        python check.py
        
    - name: commit & push
      run: |
        if ! git diff --exit-code --quiet
        then
          git config user.name github-actions
          git config user.email [e-mail]
          git add .
          git commit -m "Update"
          git push
        fi

The expression '*/10 0-8 * * *' means running it every 10 minutes between 9 AM and 5 PM Japan Standard Time. Note that this must be written in UTC, not JST.
Also, pay attention to the GitHub Actions usage limits. Currently, the free tier provides 2,000 minutes/month, so plan your timing to stay within this limit.
Since this script took about 30 seconds per run, the time used per month is approximately:

0.5 (min) * 6 (times/hour) * 9 (hours) * 30 (days) = 810 (min)

Please be cautious, as your usage quota can quickly disappear if you are running various GitHub Actions across different repositories.