Numerai Compute with Google Cloud Run

日本語バージョン

Numerai Compute support

Numerai has a webhook mechanism called Numerai Compute. The official numerai-cli can be used to set up automation from webhook to forecast submission in AWS environment. numerai-cli is easy to use, but since I use GCP for model training, I wanted to integrate the automation into GCP as well, so I used Cloud Run to support Numerai Compute. For the webhook, we create an endpoint that executes methods to retrieve data, infer, and submit predictions using the Fast API, and put it in a container. The code is as follows

numerai-compute-cloudrun-template

The usage is as follows:

1. First, copy the above code and set the environment variable GCP_PROJECT to the name of your GCP project.
2. Next, register Numerai's API key with the name numerai-public-id and numerai-secret to Secret Manager.
3. Create a service account named numerai-compute for execution.
4. Grant access to cloud run/secret/artifact registry from the created service account.
5. write the code to retrieve data, infer and submit predictions in predictors/pred1/predictor.py.
6. Execute make release-pred1 in the root directory where the Makefile exists.
7. The container image is built and pushed, and the deployed URL of Cloud Run is displayed.
8. Set up a submission webhook in the Compute of Numerai model like https://YOUR_CLOUD_RUN_URL/pred/YOUR_NUMERAI_MODEL_ID.

Currently, logs are checked, stopped, and deleted from GCP's Console.

after supporting Numerai Compute.

I used to manually run the Google Colab notebook every weekend and submit predictions, but as the number of models increased, I was not assigned instances and it took more than 10 minutes to finish running everything, but now it automatically submits predictions, so I feel very I feel much better now that I have automated predictive submissions. If you haven't automated it yet, I highly recommend it!

Numerai Compute Webhook Requests

The Numerai Compute can be configured by selecting Compute from the context menu for each model, and the request sent to the URL specified in the submission webhook will be as follows. There are two differences between the test run and the actual request: the roundNumber is a string instead of a number, and the isTest item has been increased. However, since the predictive submission does not require any branching process according to the request content, there is no need to pay that much attention to it.

round At start

{
    "roundNumber": 371,
    "dataVersion: 1,
    "triggerId": "99361e43-xxxxxx-xxxxxx-xxxx-73c29c72xxxx"
}

test runtime

{
    "roundNumber": "test",
    "dataVersion": 1,
    "triggerId": 'ea6a1f34-xxxxxx-xxxxxx-xxxx-0cb7b184xxxx'
    "isTest": true
}

Request Definitions for the Fast API

The following class is defined to support the above request with the Fast API. Although it is possible to trigger a POST request without defining it this way, we do it this way because it will be validated when an unexpected request is received when operating with Cloud Run.

class NumeraiComputeWebhookRequest(BaseModel):
    roundNumber: int | str
    dataVersion: int
    triggerId: str
    isTest: bool | None = None