playwright run-server

Yusuke Iwaki 2021/07/07に更新

Playwright提供のDockerイメージには、ブラウザを実行するための各種ライブラリが同梱されているので、サーバーは npx playwright install && npx playwright run-server 8080 でポート番号8080で立ち上がる。

Herokuでもホストができそう

Dockerfile

FROM mcr.microsoft.com/playwright:focal

heroku.yml

build:
  docker:
    web: Dockerfile
run:
  web: npx -y playwright install && npx playwright run-server $PORT

2021-07-07T06:34:19.322612+00:00 heroku[web.1]: Process exited with status 1
2021-07-07T06:34:19.484348+00:00 heroku[web.1]: State changed from starting to crashed
2021-07-07T06:34:19.493027+00:00 heroku[web.1]: State changed from crashed to starting
2021-07-07T06:34:53.709848+00:00 heroku[web.1]: Starting process with command `/bin/sh -c npx\ -y\ playwright\ \&\&\ npx\ playwright\ run-server\ \52807`
2021-07-07T06:35:09.412478+00:00 app[web.1]: Usage: npx playwright [options] [command]
2021-07-07T06:35:09.412531+00:00 app[web.1]: 
2021-07-07T06:35:09.412531+00:00 app[web.1]: Options:
2021-07-07T06:35:09.412538+00:00 app[web.1]: -V, --version                          output the version number
2021-07-07T06:35:09.412539+00:00 app[web.1]: -h, --help                             display help for command
2021-07-07T06:35:09.412539+00:00 app[web.1]: 
2021-07-07T06:35:09.412540+00:00 app[web.1]: Commands:
2021-07-07T06:35:09.412544+00:00 app[web.1]: open [options] [url]                   open page in browser specified via -b, --browser
2021-07-07T06:35:09.412545+00:00 app[web.1]: codegen [options] [url]                open page and generate code for user actions
2021-07-07T06:35:09.412549+00:00 app[web.1]: debug <app> [args...]                  run command in debug mode: disable timeout, open inspector
2021-07-07T06:35:09.412550+00:00 app[web.1]: install [browserType...]               ensure browsers necessary for this version of Playwright are installed
2021-07-07T06:35:09.412554+00:00 app[web.1]: install-deps [browserType...]          install dependencies necessary to run browsers (will ask for sudo permissions)
2021-07-07T06:35:09.412555+00:00 app[web.1]: cr [options] [url]                     open page in Chromium
2021-07-07T06:35:09.412555+00:00 app[web.1]: ff [options] [url]                     open page in Firefox
2021-07-07T06:35:09.412555+00:00 app[web.1]: wk [options] [url]                     open page in WebKit
2021-07-07T06:35:09.412556+00:00 app[web.1]: screenshot [options] <url> <filename>  capture a page screenshot
2021-07-07T06:35:09.412556+00:00 app[web.1]: pdf [options] <url> <filename>         save page as pdf
2021-07-07T06:35:09.412556+00:00 app[web.1]: show-trace [options] [trace]           Show trace viewer
2021-07-07T06:35:09.412557+00:00 app[web.1]: test                                   Run tests with Playwright Test. Available in @playwright/test package.
2021-07-07T06:35:09.412557+00:00 app[web.1]: help [command]                         display help for command
2021-07-07T06:35:09.425306+00:00 app[web.1]: npm notice
2021-07-07T06:35:09.425700+00:00 app[web.1]: npm notice New minor version of npm available! 7.18.1 -> 7.19.1
2021-07-07T06:35:09.426026+00:00 app[web.1]: npm notice Changelog: <https://github.com/npm/cli/releases/tag/v7.19.1>
2021-07-07T06:35:09.426335+00:00 app[web.1]: npm notice Run `npm install -g npm@7.19.1` to update!
2021-07-07T06:35:09.426690+00:00 app[web.1]: npm notice
2021-07-07T06:35:09.523796+00:00 heroku[web.1]: Process exited with status 1
2021-07-07T06:35:09.598102+00:00 heroku[web.1]: State changed from starting to crashed
    ```

`&&` をheroku.ymlで使っても、エスケープされて、コマンドラインパラメータとしての `"&&"` になってしまうようだ。

Herokuのコンテナを起動するたびにplaywrightのバージョンが変わったりしてもいまいちなので、Dockerfileのなかで書くことにする。

```Dockerfile:Dockerfile
FROM mcr.microsoft.com/playwright:focal

WORKDIR /root
RUN npm install playwright@1.12.3 && ./node_modules/.bin/playwright install

# ./node_modules/.bin/playwright run-server

heroku.yml

build:
  docker:
    web: Dockerfile
run:
  web: ./node_modules/.bin/playwright run-server $PORT

2021-07-07T06:46:11.117086+00:00 heroku[web.1]: State changed from crashed to starting
2021-07-07T06:47:00.617866+00:00 heroku[web.1]: Starting process with command `/bin/sh -c ./node_modules/.bin/playwright\ run-server\ \33894`
2021-07-07T06:47:05.938223+00:00 app[web.1]: Listening on ws://127.0.0.1:33894/ws
2021-07-07T06:47:06.368732+00:00 heroku[web.1]: State changed from starting to up
2021-07-07T06:51:52.696404+00:00 heroku[router]: at=info method=GET path="/" host=pl-py-remote.herokuapp.com request_id=e815e3cc-e269-41e6-a07c-15721fc934c6 fwd="92.203.52.226" dyno=web.1 connect=1ms service=35ms status=200 bytes=101 protocol=http
2021-07-07T06:51:52.968215+00:00 heroku[router]: at=info method=GET path="/favicon.ico" host=pl-py-remote.herokuapp.com request_id=c24fadc2-e9d2-483f-bee3-a002672abf2d fwd="92.203.52.226" dyno=web.1 connect=1ms service=16ms status=200 bytes=101 protocol=http

起動した。

Yusuke Iwaki 2021/07/07

WebSocket経由での接続はこんな感じのを用意

main.py

from with_playwright_browser import playwright_browser
from playwright.sync_api import Page

with playwright_browser('chromium', headless=False) as browser:
    page: Page = browser.new_page()
    page.goto('https://github.com/YusukeIwaki')
    page.screenshot(path='YusukeIwaki.png')

with playwright_browser('firefox', headless=False) as browser:
    page: Page = browser.new_page()
    page.goto('https://github.com/microsoft')
    page.screenshot(path='microsoft.png')

こんな感じのテストスクリプト。

Macローカルで

DEBUG=pw:* npx playwright@next run-server 8080 でサーバーを起動
DEBUGP=1 python main.py でスクリプトを実行

すると、WebSocket経由で自動操作できる。

ローカルでの動作確認ができたところで、WebSocketのURLをHerokuに変える。

headless=False（=headfulモード）のままだと

playwright._impl._api_types.Error: Protocol error (Browser.getVersion): Browser closed.
==================== Browser output: ====================
<launching> /ms-playwright/chromium-888113/chrome-linux/chrome --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disable-background-timer-throttling --disable-backgrounding-occluded-windows --disable-breakpad --disable-client-side-phishing-detection --disable-component-extensions-with-background-pages --disable-default-apps --disable-dev-shm-usage --disable-extensions --disable-features=TranslateUI,BlinkGenPropertyTrees,ImprovedCookieControls,SameSiteByDefaultCookies,LazyFrameLoading,GlobalMediaControls --allow-pre-commit-input --disable-hang-monitor --disable-ipc-flooding-protection --disable-popup-blocking --disable-prompt-on-repost --disable-renderer-backgrounding --disable-sync --force-color-profile=srgb --metrics-recording-only --no-first-run --enable-automation --password-store=basic --use-mock-keychain --no-service-autorun --user-data-dir=/tmp/playwright_chromiumdev_profile-corTqK --remote-debugging-pipe --no-sandbox --no-startup-window
<launched> pid=116
[pid=116][err] [116:116:0707/104847.413393:ERROR:browser_main_loop.cc(1412)] Unable to open X display.
=========================== logs ===========================
<launching> /ms-playwright/chromium-888113/chrome-linux/chrome --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disable-background-timer-throttling --disable-backgrounding-occluded-windows --disable-breakpad --disable-client-side-phishing-detection --disable-component-extensions-with-background-pages --disable-default-apps --disable-dev-shm-usage --disable-extensions --disable-features=TranslateUI,BlinkGenPropertyTrees,ImprovedCookieControls,SameSiteByDefaultCookies,LazyFrameLoading,GlobalMediaControls --allow-pre-commit-input --disable-hang-monitor --disable-ipc-flooding-protection --disable-popup-blocking --disable-prompt-on-repost --disable-renderer-backgrounding --disable-sync --force-color-profile=srgb --metrics-recording-only --no-first-run --enable-automation --password-store=basic --use-mock-keychain --no-service-autorun --user-data-dir=/tmp/playwright_chromiumdev_profile-corTqK --remote-debugging-pipe --no-sandbox --no-startup-window
<launched> pid=116
[pid=116][err] [116:116:0707/104847.413393:ERROR:browser_main_loop.cc(1412)] Unable to open X display.

Xが起動できねーと言われる。headlessモードに変更すると...

2021-07-07T10:51:26.152122+00:00 heroku[router]: at=info method=GET path="/ws" host=pl-py-remote.herokuapp.com request_id=93dc4816-6353-4baf-814f-d58a303647a5 fwd="92.203.52.226" dyno=web.1 connect=1ms service=4640ms status=101 bytes=129 protocol=https
2021-07-07T10:51:35.566919+00:00 heroku[router]: at=info method=GET path="/ws" host=pl-py-remote.herokuapp.com request_id=a8c3f165-f7e3-4415-adf1-275730467649 fwd="92.203.52.226" dyno=web.1 connect=0ms service=8423ms status=101 bytes=129 protocol=https

ちゃんと疎通した!!

Yusuke Iwaki 2021/07/08に更新

Alpine LinuxにPlaywrightを入れてみる。

公式には、できないと言われている。

$ docker run --rm -it python:3.9-alpine sh
/ # pip install playwright==1.12.0
ERROR: Could not find a version that satisfies the requirement playwright==1.12.0 (from versions: 0.0.1, 0.0.2, 0.0.3, 0.0.4, 0.0.5, 0.0.6, 0.0.7, 0.0.8, 0.0.9, 0.0.10, 0.0.11, 0.0.12, 0.0.13, 0.0.14, 0.0.15, 0.0.16, 0.0.17, 0.0.18, 0.7.0, 0.7.1, 0.8.0, 0.140.0, 0.142.0, 0.142.1, 0.142.2, 0.142.3, 0.150.0, 0.151.0, 0.151.1, 0.152.0, 0.162.0, 0.162.1, 0.162.2)
ERROR: No matching distribution found for playwright==1.12.0

うん、確かにできない。

ただ、Gitから直接入れたら...

/ # apk add git
fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/community/x86_64/APKINDEX.tar.gz
(1/5) Installing brotli-libs (1.0.9-r5)
(2/5) Installing nghttp2-libs (1.43.0-r0)
(3/5) Installing libcurl (7.77.0-r1)
(4/5) Installing pcre2 (10.36-r0)
(5/5) Installing git (2.32.0-r0)
Executing busybox-1.33.1-r2.trigger
OK: 27 MiB in 41 packages
/ # pip install git+https://github.com/microsoft/playwright-python@v1.12.0
Collecting git+https://github.com/microsoft/playwright-python@v1.12.0
  Cloning https://github.com/microsoft/playwright-python (to revision v1.12.0) to /tmp/pip-req-build-zym0dcz0
  Running command git clone -q https://github.com/microsoft/playwright-python /tmp/pip-req-build-zym0dcz0
  Running command git checkout -q 7b920eaad9ca1b53e0bc19876929c7a4d8d9a6e1
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting greenlet>=1.0.0
  Downloading greenlet-1.1.0.tar.gz (85 kB)
     |████████████████████████████████| 85 kB 763 kB/s 
Collecting websockets>=8.1
  Downloading websockets-9.1.tar.gz (76 kB)
     |████████████████████████████████| 76 kB 944 kB/s 
Collecting pyee>=8.0.1
  Downloading pyee-8.1.0-py2.py3-none-any.whl (12 kB)
Building wheels for collected packages: playwright, greenlet, websockets
  Building wheel for playwright (PEP 517) ... error

  (中略)

  gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -DTHREAD_STACK_SIZE=0x100000 -fPIC -I/usr/local/include/python3.9 -c src/greenlet/greenlet.c -o build/temp.linux-x86_64-3.9/src/greenlet/greenlet.o
  error: command 'gcc' failed: No such file or directory
  ----------------------------------------
  ERROR: Failed building wheel for greenlet
  Running setup.py clean for greenlet
  Building wheel for websockets (setup.py) ... done
  Created wheel for websockets: filename=websockets-9.1-cp39-cp39-linux_x86_64.whl size=85895 sha256=5a171732a423265dcd0328625b70b595a88ba1d5bed174417a6c76caad1934d4
  Stored in directory: /root/.cache/pip/wheels/57/c2/6d/480a9ce5f2007b3ef1b1ca384b382280209c61be58fc8994c3
Successfully built websockets
Failed to build playwright greenlet
ERROR: Could not build wheels for playwright which use PEP 517 and cannot be installed directly

gccが必要らしい。build-baseを追加でインストールすると、今度は

  adding 'playwright-1.12.0.dist-info/entry_points.txt'
  adding 'playwright-1.12.0.dist-info/top_level.txt'
  adding 'playwright-1.12.0.dist-info/RECORD'
  removing build/bdist.linux-x86_64/wheel
  Fetching  https://playwright.azureedge.net/builds/driver/next/playwright-1.12.0-1623185968000-linux.zip
  error: [Errno 2] No such file or directory: 'curl'
  ----------------------------------------
  ERROR: Failed building wheel for playwright
  Building wheel for greenlet (setup.py) ... done
  Created wheel for greenlet: filename=greenlet-1.1.0-cp39-cp39-linux_x86_64.whl size=89913 sha256=fda40b35aa230672a62b7053dabea03e41bc8fb561107e55fb8ac45c987b0822
  Stored in directory: /root/.cache/pip/wheels/79/88/67/99d62e1ad322548af24e31f462acc567159cf0f859e5b70f81
Successfully built greenlet
Failed to build playwright
ERROR: Could not build wheels for playwright which use PEP 517 and cannot be installed directly

curlも必要らしい。curlも追加でインストールすると...

  Stored in directory: /tmp/pip-ephem-wheel-cache-bk3cgbtn/wheels/d5/97/4c/e2fdde69a683ee99b1ceb360c75b01d8f29b056d484ba51962
Successfully built playwright
Installing collected packages: websockets, pyee, greenlet, playwright
Successfully installed greenlet-1.1.0 playwright-1.12.0 pyee-8.1.0 websockets-9.1
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

いけた！

まとめると、以下のようなDockerfileを作っておけばよさそう。

FROM python:3.9-alpine

RUN apk add --no-cache --virtual .install-deps build-base curl git \
    && pip install git+https://github.com/microsoft/playwright-python@v1.12.1 \
    && apk del .install-deps

このスクラップは2021/07/07にクローズされました