[M1 Mac] Setting up a local environment and essential tools for ISUCON 11 qualify problems

matsuu/cloud-init-isucon: cloud-config collection for building ISUCON past exam environments

I used the above repository to build a local environment for isucon11-qualify. In addition, I introduced some tools that seem to be the bare minimum required, so I'm leaving this as a memo.

Please note that I am a beginner who is planning to participate in ISUCON for the first time this year.

Environment

• M1 MacBook Air (16GB RAM)

Local Environment Setup

Install multipass

mac

brew install multipass

Clone the repository and build the Ubuntu environment

Clone the repository and launch an Ubuntu instance using the multipass command. Set the memory to 4GB.

mac

git clone https://github.com/matsuu/cloud-init-isucon
cd cloud-init-isucon/isucon11q 
multipass launch --name isucon11q --cpus 2 --disk 16G --mem 4G --cloud-init isucon11q.cfg 20.04

Wait for a while.

matsuu/cloud-init-isucon: cloud-config collection for building ISUCON past exam environments

Since cloud-init takes time, a message like the following may be displayed, but the build is continuing in the background.

As mentioned, don't panic even if a timeout error appears; just wait.

Once the build is complete,

mac

multipass shell isucon11q

You can enter the launched Ubuntu instance with the command above. Once inside,

ubuntu

sudo -i -u isucon

Switch to the isucon user. Basically, all subsequent tasks on Ubuntu will be performed as the isucon user.

Change Language

isucon11-qualify/manual.md at main · isucon/isucon11-qualify

Change the programming language according to the manual. This time, we will set up Ruby.

ubuntu

sudo systemctl disable --now isucondition.go.service
sudo systemctl enable --now isucondition.ruby.service

Try Running the Benchmark

Confirm that the bench command can be executed.

ubuntu

cd bench
./bench -all-addresses 127.0.0.11 -target 127.0.0.11:443 -tls -jia-service-url http://127.0.0.1:4999

Try Opening the Application in a Browser

Run the following command:

mac

multipass ls

You will get the following output:

mac

Name                    State             IPv4             Image
isucon11q               Running           ***.***.***.***     Ubuntu 20.04 LTS

Check the IPv4 value and open http://***.***.***.***:3000 in your browser. You should be able to see the ISU CONDITION app.

Setting Up Useful Tools

Enabling Connection via VSCode Remote Development

This assumes you use VSCode as your editor and have already installed the VSCode Remote Development extension.

Setting up remote access via VSCode will make various tasks easier.

SSH Configuration

First, create an SSH key on your Mac. Name it id_rsa_isucon11q.

mac

cd ~/.ssh
ssh-keygen -t rsa -f id_rsa_isucon11q

Take note of the public key value.

mac

cat id_rsa_isucon11q.pub

Create ~/.ssh/authorized_keys on the Ubuntu instance and add the public key value to it.

ubuntu

cd ~/.ssh
touch authorized_keys
chmod 600 authorized_keys
vi authorized_keys # Open with vi editor, paste the public key value, and save

Verify that you can SSH from your Mac terminal.

mac

ssh isucon@***.***.***.*** -i ~/.ssh/id_rsa_isucon11q

VSCode Configuration

From the VSCode left menu, select Remote Explorer -> SSH Targets -> + button, and enter the SSH command you just used to verify the connection: ssh isucon@***.***.***.*** -i ~/.ssh/id_rsa_isucon11q.

Using Open Folder to select / (root) seems convenient for overlooking all the files at once.

Introducing alp

Install alp on the Ubuntu instance.

Check the URL for the file you want to download from Releases · tkuchiki/alp and use the wget command to download it.

ubuntu

wget https://github.com/tkuchiki/alp/releases/download/v1.0.9/alp_linux_arm64.tar.gz
tar -zxvf alp_linux_arm64.tar.gz

Referring to installation,

ubuntu

sudo install alp /usr/local/bin/alp

Run the command above to install it.

ubutnu

alp --version

Verify that it works.

Nginx Log Configuration

To analyze Nginx logs with alp, you need to set the output format to either:

• LTSV
• JSON

In this case, we will set up LTSV.

ubuntu

sudo vi /etc/nginx/nginx.conf

/etc/nginx/nginx/conf

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

-   log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
-                     '$status $body_bytes_sent "$http_referer" '
-                     '"$http_user_agent" "$http_x_forwarded_for"';

-   access_log  /var/log/nginx/access.log  main;

+  log_format ltsv "time:$time_local"
+     "\thost:$remote_addr"
+     "\tforwardedfor:$http_x_forwarded_for"
+     "\treq:$request"
+     "\tmethod:$request_method"
+     "\turi:$request_uri"
+     "\tstatus:$status"
+     "\tsize:$body_bytes_sent"
+     "\treferer:$http_referer"
+     "\tua:$http_user_agent"
+     "\treqtime:$request_time"
+     "\truntime:$upstream_http_x_runtime"
+     "\tapptime:$upstream_response_time"
+     "\tcache:$upstream_http_x_cache"
+     "\tvhost:$host";

+   access_log  /var/log/nginx/access.log  ltsv;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*.conf;
}

Modify the configuration file as shown above, and restart Nginx after saving.

ubuntu

sudo systemctl restart nginx

Now, let's confirm it can be analyzed with alp. First, clear the existing logs.

ubuntu

sudo truncate /var/log/nginx/access.log --size 0

Then run the bench command. After the benchmark is complete, analyze the Nginx logs with alp.

ubuntu

sudo cat /var/log/nginx/access.log | alp ltsv

MariaDB Slow Query Log Configuration

Configure the MariaDB slow query log. I'm not very familiar with it, but since I understand it's compatible with MySQL, I'll proceed with the setup while referring to MySQL information as well.

ubuntu

sudo vi /etc/mysql/conf.d/my.cnf

/etc/mysql/conf.d/my.cnf

[mysqld]
character-set-server=utf8mb4
+slow_query_log=ON
+long_query_time=0.001
+slow_query_log_file=my-slow.log
[mysql]
default-character-set=utf8mb4
[client]
default-character-set=utf8mb4

By the way, with this configuration, the slow query log seems to be output to /var/lib/mysql/my-slow.log.

Restart to apply the settings.

ubuntu

sudo systemctl restart mysql

Introducing pt-query-digest

Since pt-query-digest is included in percona-toolkit, we will install it.

ubuntu

sudo apt install -y percona-toolkit

Let's check if it works. After running the bench command,

sudo pt-query-digest /var/lib/mysql/my-slow.log > /tmp/pt-query-digest.txt

Run the command above. Then check /tmp/pt-query-digest.txt to confirm that the report results have been output.

References

• matsuu/cloud-init-isucon: cloud-config collection for building ISUCON past exam environments
• Local build memo for isucon 11 qualifying environment
• How to use alp (Basics)
• Preparing to participate in isucon7 - Qiita
• [Linux] Methods for compressing and decompressing files - Qiita
• Creating a file with a specified name using ssh-keygen - Qiita
• Summary of SSH public key authentication settings - Qiita
• Do not use xxx.xxx.xxx.xxx for IP address examples - Qiita
• Should you not use "xxx..." for IP address examples? "I thought it was an annoying sermon, but it wasn't" and other reactions - ITmedia NEWS
• It's convenient to name virtual machines created with multipass - Qiita
• [Ruby] Tracing isucon11-qualify while looking at explanations
• MySQL Slow Query Improvement for Beginners
• pt-query-digest when data increases in MySQL - Qiita
• percona/percona-toolkit: Percona Toolkit
• Quickly identifying the cause of high DB load in seconds (aws RDS) - Qiita
• Environment setup for isucon8 qualifying - Analysis edition - Qiita
• Linux commands to empty an existing file - Qiita