【Optimization Method】Optuna Tutorial part2

This is part 2 of the Optuna tutorial series, please check the previous if you haven't checked.

Official Page:
https://optuna.org/
Official Tutorial:
https://optuna.readthedocs.io/en/stable/tutorial/index.html

2.0 Review the part 1 code

import optuna

def objective(trial):
    x = trial.suggest_float("x", -10, 10)
    return (x - 2) ** 2

study = optuna.create_study()
study.optimize(objective, n_trials=100)

best_params = study.best_params
found_x = best_params["x"]
print("Found x: {}, (x - 2)^2: {}".format(found_x, (found_x - 2) ** 2))

2.1 Search Space

For hyperparameter sampling, Optuna provides the following features:

• For categorical params: optuna.trial.Trial.suggest_categorical()
  ex. ["cosine", "onecycle", "cosine_with_warm", ...]
• For integer params: optuna.trial.Trial.suggest_int()
  ex. [-100 ~ 100]
• For floating point params: optuna.trial.Trial.suggest_float()
  ex. [-100.50 ~ 100.50]

With optional arguments of step and log, we can discretize or take the logarithm of integer and floating point parameters.

import optuna

def objective(trial):
    # Categorical parameter
    optimizer = trial.suggest_categorical("optimizer", ["MomentumSGD", "Adam"])

    # Integer parameter
    num_layers = trial.suggest_int("num_layers", 1, 3)

    # Integer parameter (log)
    num_channels = trial.suggest_int("num_channels", 32, 512, log=True)

    # Integer parameter (discretized)
    num_units = trial.suggest_int("num_units", 10, 100, step=5)

    # Floating point parameter
    dropout_rate = trial.suggest_float("dropout_rate", 0.0, 1.0)

    # Floating point parameter (log)
    learning_rate = trial.suggest_float("learning_rate", 1e-5, 1e-2, log=True)

    # Floating point parameter (discretized)
    drop_path_rate = trial.suggest_float("drop_path_rate", 0.0, 1.0, step=0.1)

2.2 Branches and Loops

The ~optuna.trial.Trial object enables to be applied branches and loops as is. It provides development experiments like usual.

Branches

import sklearn.ensemble
import sklearn.svm


def objective(trial):
    classifier_name = trial.suggest_categorical("classifier", ["SVC", "RandomForest"])
    if classifier_name == "SVC":
        svc_c = trial.suggest_float("svc_c", 1e-10, 1e10, log=True)
        classifier_obj = sklearn.svm.SVC(C=svc_c)
    else:
        rf_max_depth = trial.suggest_int("rf_max_depth", 2, 32, log=True)
        classifier_obj = sklearn.ensemble.RandomForestClassifier(max_depth=rf_max_depth)

Loops

*The variables that outside of an objective function are also could be optimized of course.

import torch
import torch.nn as nn


def create_model(trial, in_size):
    n_layers = trial.suggest_int("n_layers", 1, 3)

    layers = []
    for i in range(n_layers):
        n_units = trial.suggest_int("n_units_l{}".format(i), 4, 128, log=True)
        layers.append(nn.Linear(in_size, n_units))
        layers.append(nn.ReLU())
        in_size = n_units
    layers.append(nn.Linear(in_size, 10))

    return nn.Sequential(*layers)

2.3 Notes

The difficulty of optimization increases roughly exponentially with regard to the number of parameters. That is, the number of necessary trials increases exponentially when you increase the number of parameters, so it is recommended to not add unimportant parameters.

2.4 Summary

1. Optuna has three types of ways to specify the variables that would be optimized, int, float, and categorical. And we can specify the attribution of the step and log as necessary.

2. The Trial object cloud be treated within branches and loops as is.

The Optuna is a user-friendly library.
Part 2 has ended. I'll write the part 3 soon.